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FORETORD 



This pamplilet is based on tl\e experience a number of 
Commission staff over the years in pre^ari ng ^est ionnaire!9 as 
well as on the available literature. The present edition was 
prepared by Pauline A. Duckworth. 




William A. Gorham, Director 
Personnel Research and 
Development Center 
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INTRODUCTION 
Purpose of the Pamphlet 



The use of questionnaires in business, industry, government, and 
education has increased steadily In recent years. They are used in 
many different areas of government operation, including job analyses, 
mail investigation of candidates^ qualifications, research on the 
effectiveness of testing instruments, and attitudinal and informational 
surveys. 

Many people who associate the use of the questionnaire method 
with certain well-known fields such as public opinion polling, attitudinal 
research, radio- te levision studies, and census operations do not realize 
how much broader is the actual application of the questionnaire method. 

There are many types of questionnaires. For example, application 
for any position normally requires' the candidate to complete a form 
describing his education, training, and experience. This form is one 
type of questionnaire. A supervisor who is sent a form which asks him 
to report on or to evaluate a former employee's job efficiency receives 
a different kind of questionnaire, a qualifications inquiry. Again, an 
employee may be asked to indicate on a checklist the duties of his job 
which he considers the most or the least important. This checkl,ist is 
also a questionnaire. 

In the ps^ychological literature many books / art ic les , and pamphlets 
deal either. direct ly or indirectly with the development of questionnaires 
as one kind of testing instrument. The available material- is often geared 
to a specific class of users- - public opinion surveyors, educators, and 
statisticians, for example. The procedures and problems described in 
such material are often fairly specific to one type of survey and are 
meant for a reading audience' with knowledge of testing. 

Since the use of questionnaires in government is actually as well 
as potentially broad, and because all questionnaires are designed on the 
basis of certain common principles, a pamphlet of instructions on the 
construction of questionnaires in general may provide a simple, concrete 
guide to the development of new questionnaires and the improvement of 
those already in use. 

Many government people who are interested in obtaining information 
by the questionnaire method are not specialists in measurement procedures, 
but are nevertheless interested Iji collecting the necessary data , in ^ 
accordance with the most effective procedures. We hope that these people 
willviind this pamphlet helpful in their planning for, and design of, 
questionnaire forms. Administrators arid reviewers may find it useful 
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in r-valuaiing proposed ques L ionnairos or those already in use, and 
experienced sp«,'C i/i li s Ls may find it convenient to have the basic steps 
in questionnaire construction outlined in a simple fashion in one ^ 
pamphlet. 

The emphasis in the pamphlet is on the basic qualities which 
questionnaires must have in order to serve best the purpose for which 
they are'designed. A description of the preliminary planning necessary • 
to questionnaire construction, as well as the construction of the form 
itself, is providf'd. Various types of questionnaire i tems and the 
advantages and disadvantages of each are discussed and illustrated, as 
are instructions' in effective item-writing techniques. 

Some readers mav , perhaps, wonder why we have placed so much stress 
on the manv pitfalls that await the unwary constructors of questionnaires 
In uKplanation, we offer the comment of Bruce Barton, a leading figure 
in the advertising business for many years and a two- term member of the 
House of Representatives. Mr. Barton once remarked that "Most people 
will frankly admit that they can't direct a dance orches tra— that . they 
wouldn't know how to operate for appendicitis— but almos. everyone seems 
to feel that he (or she) can work up an effective questionnaire." 
Because Mr. Barton's statement is, unfortunately, all too true, we 
have tried to increase the questionnaire constructor's awareness and 
understanding of the problems and ^hazards inherent in questionnaire 
design and consLruction. 
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CHAFVER 1 

Character! St ICS of a Good Quos L i onnai i^o 

A questionnaire is a testing .instrument in written form v;hose 
purpose is to obtain information about the subject matter with which 
..the questionnaire is concerned. We must apply to the development of 
questionnaires 'the s. principles which apply to the development of 
tests in general. Years of research in the proper construction and use 
of testing instruments h^uve r.hor/n that there are great differences in 
the effectiveness of "go6d" and "bad" ins t ranlen t s . To be "good" a 
testing instrument, such as a questionnaire, must be properly designed 
and used. 

A. Validity 

First, it must accomplish the purpose for which it is designed. 
A questionnaire designed to assist in the selection of people for a 
job, for example, must demonstrate its validity by separating those 
who are likely to succeed from those who are unlikely to succeed in a 
given situation. The better the instrument, the finer the separations 
or discriminations it makes, 

A questionnaire deve loped .for a survey of attitudes toward -public 
issues is considered valid if there is statistical proof that .the re > 
spondents expressed attitudes representing their true opinions at that 
particular time and that the particular sample of 'Respondents was 
representative of the total population. Once this proof is established, 
conclusions can be drawn about the actitrdes of the larger, population 
represented by the sample. 

B. ^ Reliability - 

Second , a good testing instrument must measure candidates ' abilities 
in a consistent, manner. A ques t ionnaire ' s ' re li abi 1 i t y is measured in 
terms of the accuracy or consistency of agreement of both respondent 
and rater. Respondent reliability is achieved if respondents reply 
in the same fashion when a questionnaire is submitted to them a second 
time, or if they respond in much the same way to an alternate form of 
que?^ tionnaire which is equivalent to the first one. 

?.ater reliability is achieved if two independent raters score the 
questionnaire in the same way, or if one rater scores it in much the 
same way at different times. On careful ly ^circumscribed questions 
with limits put ° on poss ib le responses, we would expect high rater 
reliability. On the other ha" id , items that permit free answers do 
not lend themselves either to high rater reliability or to high 
respondent reliability. 
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The reliabilicy of many kinds of government: questionnaires cannot 
be measured statistically. One example; is a questionnaire sent to peopl 
on a register, to determine their availability for a particular job; The 
first time a person fills out the form, he may indicate his availability 
but: not be selected for the position; the next time he is asked, he may 
say that he is not available. In this illustration, inconsistency of 
the responses given at different times does not reflect unreliability 
of! the question or of the ins t rumen t i t merely shows that the person's 
status or attitude has^ changed. The questionnaire is, therefore, 
-furnishing valid information which is reliable in the sense that it 
is accurate. 

Administration , and Scoring 

Third, the practicality of admijnistration and scoring -must be con- 
sidered in the design of any good questionnaire. Whether the form is 
administered to a group, whether it is filled out by an interviewer, 
or whether it is sent through the mail, it must be easy to use. The 
ease with which a form is used depends on a number of factors, e.g., 
physical appearance of the form itself, tlie arrangement or order of the 
questions, clarity of directions, and the form or type ( f ree- answei' , 
multiple-choice, etc.) of ^ the questions. Confusing and complicated 
directions or a jumbled appearance will tend t o 'd iscoui^age -espondent 
cooperation. However, whei^ the form appears to be simple to complete, 
a respondent is more likely, to respond carefully and, consequently, 
more accurately. It is alsCt true that in using a form which is con- 
fusing to follow, an interviev;er may have d\fficul.ty establishing, or 
maintaining good rapport with the interviewee. 

The length of a questionnaire merits some consideration. On the 
whole, we should not ask a respondent to spend too much of his time in 
completing a questionnaire. However, fairly extensive questionnaires , 
if they are carefully constructed and laid out, receive very satisfactory 
returns . -.^ ^ , 

The type of paper used for questionnaires varies v^idely. ^. Some are 
printed on expensive paper; others\are mimeographed or otherwise 
inexpensively reprofiuced on cheap pVper. The type of paper and method 
of reproduction depend upon the pur^pse of the questionnaire, th,e si^e 
of the group to which it will be administered, financial resources ■■ 
available for the study, etc. 

Ease of scoring or rating respons'es is important. The form of the 
question used affects this characteristic of questionnaires to a large 
extent. Different item types will be c^iscussed 'later in detail. It 
. is sufficient here to say that the questionnaire itself will be more 
manageable if possible answers can be art^^anged in accordance with a 
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coding system decided upon in advance. If such a system is employed, 
classification and collection of data can be a relatively simple operation. 
Furthermore, a careful and complete definition of the use to which the 
data will be put and the method of analysis to-be employed will greatly 
simplify scoring or rating responses.- ■ , 

Face Validity 

Fourth, we oust determine whether, in a particular survev situntion, 
a questionnaire constructor should be concerned with face validity. In 
the government, eKamining instruments should usually not onlv bu valid 
but should also seem practical and reasonable to the competitors in 
view of the job duties and responsibilities. 

For most effective results, a respondent ,nust be motivated to reply 
with care, and to the best of his knowledge. An instrument which appears 
to him to contain questions related to the ' particular goal istan incen- 
tive to do his b.;St. 

There are, however, situations in which the goal is such that the 
questionnaire constructor may wish to avoid face validity. For example, 
a questionnaire of a direct type developed to find out employees' atti-' 
tudes toward their supervisors rnay evoke unreliable answers. The employee 
uncertain that his replies will be kept confidential, may feel that his 
position will be jeopardized if he mi.:.es critical comments. Furthermore, 
as many studies have shown, people do not usually rate others low on a 
scale.. The -employee may, therefore, tend to rate his supervisor as being 
at least average in each area (lenient tendency). He may also, regard- 
less of whether he reports favorably or unfavorably, give a rating on 
.each area that reflects his overall opinion of the supervisor (halo 
effect). If a respondent is forced to make rather difficult choices, 
or if he cannot guess what the intended "right" answers^ are , he may 
feel considerable resentment. Questionnaires of this type, however, 
are developed through specialized psychological techniques and do- not 
tall Within the scope of this pamphlet. 

» 

Because of the variety of questionnaire purposes, each of these \- 
four qualities of testing instruments cannot be applied in the same 
way to all questionnaires. Constructors should, however, plan for and 
carry through an evaluation plan for the questionnaires they develop 
The matter of evaluation, as well as of data prqcessing, and adminis- 
tration of questionnai^res, will be fully discussed in a subsequent - 
pamphlet addressed to these aspects of questionnaires. 
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■ CHAPTER 2 ^ ^ 

Plans and Objectives of a Questionnaire . 

Among authors of textbooks on test construction there is_ universal 
agreement on the need for careful planning preliminary to the actual 
writing of test items and design of . the test form. Without minimizing 
the difficulty of writing items and the importance of care in their 
preparation and compilation, some specialists in test construction 
believe that the decisions mide before construction r.re likely 'to be 
more important 'and crucial than those made late.:,. 

In the course of planning, the constructor's judgments may be 
subject to revision as .he gains further information about the problem. 
Revisions in judgments may also be necessary because the planning stages 
are not .necessarily mutually exclusive nor chronologically separate. 

In making any survey, the first taskis to define exactly its 
purpose or purposes. In order to develop a clear de'finitio.ii of purpose, 
the -survey conductor should; > Nj 

I. describe 'in as much detail as possible t .e overall problem 
to be investigated; 

II. decide whether a questionnaire is the prope'r tool to use in 
/ investigating the problem;- 

% Hi. define what information will be needed; 

IV. .decide to whom the questionnaire will be given; 

V. decide upon the method of administration of the questionnaire; 

VI. decide on the form the questionnaire will take and how the 
data will be analysed; . . . 

VII. consider the purpose of the survey in terms of the above 
^ JEactors; and ■• . "\. ^ 

^ ' " . 

VIII. plan for continuing documentation of the process and eventual 

*followup., 

Describe the overall problem to be investigated 

Regardless of the type of problem to be investigated, it is always 
necessary to ( 1 ) define the problem or state the general objective clearly; 
(2) obtain agreement among the persons involved in -^the study concerning 
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a definition of the problem or objective; and (3) he sure that the 
people involved in the administration of the survey understand the 
problem thoroughl.y. 

An example a statement of the general objective of a study 
might^ read as follows: 

To develop an instrument or technique by which the job performance 
of traineetv hired from a general aptitude examination may be 
evaluated. The instrument will be used in validity studies of- 
this examination. 

It may be possible at the beginning of a survey td state the 
objective very specifically, l?ut usually it can be described only in 
broad terms. However, as succeeding planning stages are considered, 
it is necessary that the objective be clearly and definitely stated. 

Decide whether a\qu es t ionnai re is the proper tool t o use in investlgat i ne 
the problem 

Before deciding to use a questionnaire, a survey conductor should 
•make certain that the information he wants can be obtained most easily 
and efficiently by a questionnaire. 

Questionnaires are used successfully to- gain information that is 
dif^ficul't or impractical to obtain through observation -or from records, 
but that can be gained by interrogation. For example, employees for a 
particular job are selected on the basis of a testing instrument.* In 
prder to determine the effectiveness of the instrument for selection; 
its validity is measured by comparing scores on .the test with success 
on the job for- a sample of employees. Successful job performance could 
be measured^by systematic observation of employees ' work and a check 
of its accuracy over^a period of time. Often, however, this, would be 
impractical, expensive, and time-consuming. A more usuaT method of 
obtaining me^hsures of job efficiency is to ask supervisors to rate ^ . *^ 
their subordinates on a form that contains items covering the ..quali ti es 
and behavior that contribute to success on the job. The supervisors' 
ratings are the criterion with which original test scores are correlated 
in order to measure test validity. The validity of the test is high if 
employees who received high test scores are rated as more proficient on 
the. job than are those who received low test scares. 

Questionnaires are also useful tools when large numbers of respondents' 
are required for a survey or study. If only a few people are needed to 
gain the desired information, a questionnaire would not be suitable be- 
cause of the- time and cost of designing it. . 

A questionnaire iJould also be inappropriate in the following 
example. An instructor wants to evaluate a reading improvemen t ' course . 
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lie could ask the participants by c}ues t i onnrj i ; 1.1: that 

th^'ir reading «^.pe'^d and comprehension had in l>e 
simplc-^r, however, to measure, by machines a\\^^ iu» purpose, 

the participants reading rate before and after the course. Comparison 
of the test results at the beginning of the course and upon its completion 
would show objectives ly the extent of improvempnt in reading speed and 
comprehension • 

De f i ne wh at in to rma t i on will be n e ed e d \^ 

\, 

Once the problem to he investigated has b^^en, described , and it has 
\)Con determined that a questionnaire is the most effective tool to use, 
the neKt stagt^ involves defining the information needed. 

Many questionnaires include certain items of information that . 
identify the respondent and the cjuestionnaire- -f or example, the 
respondent's ncime , identification number, age, present position, 
agency, and tho date and place uf administration of the questionnaire. 
OtIuMr questionnaires ask for no identification or fov partial identi- 
fication. When confidential or derogatory information may bo needed^ 
and if it is not germane to the study that the source' of the ' information 
be identified, the respondents should be told that they need not sign 
their name. 

Every questionnaire constructor sht^uld ho fully aware oi* his 
rusponsibi I i t y to the respondents t.i ensin'r th:U. ins pari ici pal i ^ n 
does n(^t work to his detriment. For example, in an employee attitude 
survey, to ask for such fine br(^akdowns of organizational units, 
biographical data, and other areas that the supervisor could readily 
identify the -responses of a particuUir individual would not be fair 
to the n^s pendent regardless of whether or not his opinions agreed 
with those of his supervisor. 

In defining the areas c^f information to be covered, the ct:)nst ructor 
slK^uld first find out whether or not there are any prcvit^us .surveys 
that would prove helpful. If such a survey is available, an evaluation 
sh >uld bo madv. of its adequacy, perhaps by consulting th(^ people who 
made the survey or checking their reports to see whether all the 
required data were collected. For example, in a survey of employee 
satisfaction, the constructor would prepare a basic list of infor- 
mation categories --adequacy of sup^irvision , physical conditions, 
adecjuacy of training, etc. --and then compare his list with the road 
artuis outlined in prtwic^us surveys of the s£\me kind. If comparable 
data are available, the constructor should use, insofar a', practicable, 
identical questions, de^f ini tions , and quantitative units of measurement 
in order that the rtjsults of his survey may be compared with ihe. 
related surveys more meaningfully. In addition, careful records of 
t^ach step in the study should be documented for future reference and 
sulist'CjUe.nt follc»wups. 
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A questionnaires const: rucLor slurjlcl noL hesitate to consult other, 
persons at any stage in the planning. If he does not have sii f f ^ c i.^n t 
knowledge to define the broad areas of the problem Ip*' Ir, -ild 
survuy relevant literature and consult people in a p^>;>iLi : . < the 

knowledge. In the previously mentioned survey of employe iides, 
he would conduct personal interviews with employees on ihv j aiiu 
with tiheir supervisors. He would also read reports of, and investigate, 
similar surveys made in industry and government. 

in any survey which attempts to measure opinions, preferences, or 
attitudes, t:he development c^f a questionnaire raises many technical 
problems and a constructor probably should consult a skill(>d research 
psychologist during the entire planning process. In many surveys made 
in the government, a tecim has the responsibility for the developnient 
and administration of the questionnaire; at least one member of the 
team sliould be an expert in the particular field in which the survey 
is being made, 

K^ien the broad categories of needed information have Ilcu deter- 
mined, they should be arranged in whatever order seems most logical 
and appropriate. A questionnaire may have to be shortened and sonu' 
categories eliminated. The constructor should ir.ake certain, however, 
that he does not confus'e conciseness with brevity by eliminating 
iniportant categories whose exclusion will seriously impair the 
qu(ist ionnai re ' s effectiveness. 

Decide to whom the gue s t ionnai V- will be g i v e n 

Dt'ciding to what group the questionnaire will be given necessitates 
some knowledge of the characteristics and size of the group as well as 
of sampling procedures. 

The purpose of the survey generally determines the compt>sition of 
the group to be surveyed. Thi s ,de t erminat i on can be made after investiga- 
tion of the total possi'ble number and kind of people v^ho could participate 
in the survey. In many cases it is not practical to use the total possible 
group. To illustrate, a survey of morale- in a small field office could 
include the entire staff of the office, whereas the same type of survey 
conducted in a large central office probably would require selecting a 
sample of employees. 

Basically, a sample is selected so that it is representative of 
the tc^tal "population." The total populaticm consists of all the people 
wh > possess the characteristics pertirjent to a particular study. The 
sample must be representative in orde" that the conclusions drawn and 
the predictions made on tha; basis of the sample results will apply not 
only t(^ tin* sample group but. also to the larger population which it 
reprf\sont s . 
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A rt'proj;LMU.al.ive sample duplical:os the character i s L ics of the total 
population that aro Likely to influence the results of the survey. The 
characteristics p.ocessary for representativeness vary from one survey 
to another, depending on what is heing surveyed. Whether memhers of a 
sample group live in the country or in 'ity might be important in 

a survey c^f puhlic opinion and the s ■ -O-i have to contain members 

from the city and the country in t rtion that they are found 

in the total pcn^^ulation under cons i i the survey. Whether 

employees live in tlie city or the ctium i wever, would probably 

not bt! an (.\ssential characteristic in a survey of attitudes toward 
cigarette advertising. 

UfUn-inination of the characteristics of the questionnaire population 
will help provide inf o rmat it:tn necessary for actual item cons t ruct. ion . 
As be seen later ^ tlie form and wording of questionnaire items are 

extruMiiely iinpcn*tant, the lengtli of the questionnaire may affect i.ts 
usefulness, and proper motivation of respondents Is necessary for a 
gocKl survey » Knowing what kind of people are to respond to a ques- 
tionnaire is essential to the development of it. 

The techniques of pmper sampling are complicated, and it is 
essential f.(^ r anyone who is not skilled in sampling tecliniques and 
procedures to obtain te^Tlnical assistance, 

Cinct- th<» sampling tt^chnique has been decided, a list of the steps 
lo he followt'd sliould be pi^epared for tlie guidance of those who will 
par ti c ipat e , 

Dec icK- upon t he me thud of adi:ii n 1 st rat icm of I he quest ionna i re 

Ooo melliod uf adiiiini s t ra't ion involves direct contact with the 
respondi^nl in a personal interview. E\n-sonal interviewing with a 
■questionnaire form is a widely used technique in puhlic opinion polling, 
radio and IV survi^ys, and (jualif ications invos t i t- ions . The interviewer 
may tisr' a s tandardi^itnl cjues l i cmnai re form, asking the respondent each 
cjuestion on the form and recording liis answers. He may, instead, conduct 
a "ff^cused" interview. Such an interview consists of a series t)f open- 
end or f;;ee- answer questions ct^ntering about tliu questionnaire topic. 
These rjuestions are thcuight out in advance by the interviewer and he 
rc^cords the rt^spondent ' s answers as nearly verbatim as possible. The 
fcuMu .i>f t lu' focust'tl interview is fairly uns t rue t ured- - t ha t is, the 
i nt t> rv iov;eM' is giv(:jn considerable leeway in his dealings with the- 
ri'.S'poncion t . An example of an open-end question is, "UTiat do you tliink 
.ihoiit the clly's >:oning laws?" Depending upon the. reSpond(>n t ' s reply, 
[hv int tM*vii'Wer may on to otlier (juestitms related to the quc>st ionna i re 
U^pic or hi' iiriv ask further cjuestions about tlie zoning laws in order 
to oh t ran iiior(» ci^iiiph't.o information about thv res[)cmdent ' s c^pinions. 

A second possible method of ohlaining completed questionnaires is 
to inai 1 t ht i'orin t o r<.'Sponcb'n t s . 
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Finally, questionnaires may bc-^ administered direcLly to a group, 
either assembled in one place or in their usual places of work, 

A decision as to the best method of administration of a questionnaire 
can be made only after a review of the advantages and disadvantages of 
each method. ^ The following evaluations of the methods are based on tho 
methods themsalves, assuming optimuio \ '^rdi nf^ of the items and optimum 
admini s t rative condi tions , 



Advai . 



of the personal 



1, 



2. 



It yields a high proportion of usable, coniphUcd questionnaires 
that cover the required sample population. 



Bwer to clear up any misunderstanding or 

n* rephraj.ing 



It pe rmits the interviev. _. _ _. _ 

misinterpretations of the Intervievv'ee by repeating 
questions , 



3. It enables the interviewer to secure spontaneous rather than 
studied responses to the questions he asks, _ 

4. It may provide confidential" or derogatory information whic 



respondent would n> 
about which the -f- 
sandwiched between 
hav(.» an emotional 
give in writing cc 
give orally. 



otherwise fui'n: ^^h, 
:dent is lik- 1 

.s t i 6ns to ^ li 
ction, Snm*^ p' OT 

;i kinds of : n for 



^^uestion ^ on sub 

sensitive may 
;i 1 1 probably no 
. too , h'^s i I: ate 1 1 - 
ion which they wi 



Disadvantages of the pers 



L nterview; 



The less structure there is in the form used by. the interviewer, 
the more difficuM. it is to analyze the responses statistically. 
The variety of responses given to free-answer questions does 
not always lend itjr'lf to coding methods which facilitate 
ana lys i s , 



The reliability (^f 

i rv erview is IV- 
f : lied out \ -.v 
■J^plaining 

interviewee . :,iy 

responses; a reir 
repl ies , 



i cue . t ionnai re com: 
V- be lower than t 
■ s pondent himse 1 f , 
adapt ing his lar 
lously or unconsc: 



^■ted during a person 1 
reliability of a f rii 
interviewer 5 in 
aage to the particul r 
ously affect the latt'^r's 



. ew by another persc^n might yield' different 



Persona 1 in terviews 
and t ransport at ion c 
t rai nod i n t e rv i ewe r : 
properly selecting.. 



re very expensive. They are timo-consuming, 
sts are high. They require a staff of 
as well as an organization capable of 
raining, and supervising the interviewers. 
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4, Personal interviews take more of the respondent's time than 

other methods. If the respondent does not have sufficient time 
for the interview and a return visit on the part of the inter- 
viewers is necessary, the general cost of the interview is 
increased. 

Advantages of mail questionnaires: 



They permit a broader geographical ^ fa and a larger number of 
respondents to be covered at l- /vpense than is possible with 
personal interviews. This lo\\ cost will not necessarily apply 
if personal followups are made. 

The expense and time necessary to insure a staff of trained 
interviewers ai'e eliminated. Much of the work involved in 

done by a clerical staff. 



mailing inquiries can 

3, The questions are sta* 

4, The respondent can cr 
take as much time as 

Disadvantages of mail question: 

1. Many mail inquiries d^e 
of 'hi as in the results 
who »tur'n the fonis n.: 
sanii . group , 

2, Man> mail inquiries f-i 
incomplete or becaust 
St and able. 



f O: 



ill resp 



jn t s , 



form at ^.is convenience and 
o fill i.z out , 



urned . Thi s rai ss the pr;^b lem 
irvey because the respondents 
repreSifintative of the lotal 



.;)le5 either because they are 
.ponses are not legible or under- 



6. 



The I' ngth and appear, 
its eiiectiveness , A 
long tenu^ tr discoura 

Respondents may not p i ^: 
information , 

Many questions which miLhi 
from mail questionnaires. 

A questionnaire may be i 
to whom it is addressed : 



Despite the disadvantages of mail 
out that the response' rate to mar 
depend s upon a comb inat ion of f .-iv 
and quality of the questionnair 
letter , etc . 



h mail questionnaire car fluence 
:ich looks LOO complicate too 
etion and return, 

c nfidential or derogatory 

a' "agonize respondents must he omitted 



: by someone other than the person 
: /.ample , bv a secretary. 

onnaires cited, we should point . 
lUMu is good. It undoubtedly 
rhr group surveyed, the relevance 
^fTsuasivenoss of the covering 
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Advantages of quesCionnciires administered either r.o a group assembled in 
one place, or to persons at their individual workplaces: 

1. l^Hien respondents are assembled in one place, the physical 
conditions are controlled and standardized for all. 

2. One or two persons can supervise the administration of the 
questionnaire to large groups, and only a small staff of 
trained people is needed. 



a 

is an 



3. Respondents may cooperate more fully if they are part of e 
group and know that administration of the questionnaire is. 
approved procedure for which time has been allotted, 

4; All questionnaires are returned and are likely to be completely 

filled out because- respondents can ask questions of administrators. 

5. Questionnaires which are filled out by the res^pondents at,.their 
normal , places of work can be completed at their convenience. 
•With certain types of respondents, e.g., those who .cannot 
always conveniently leave their work or . those who work on 
predetermined time schedules, this method is ,v^'ry advantageous. 

' ' \ 

Disadvantages of above questionnaires: - " \ . 

,- ■ ' ■ ' 

1. Some questions may require the respondent to refer to records 
for his answers. Such questions would not be feasible for 
group administration, but would be satisfactory for adminis- 
tration in no r ma 1 wo^rk p laces. 

2. In group administration, it sometimes happens that not all of 
. < members" of the group can be assembled at one time. When 

such^.a "situation exists, it may be practicable to schedule 
.more than one group meeting, ^ or give : the questionnaire to 
. respondent^ at their normal workplaces. 

After consideration of the advantages and disa^^l 'antages of th^ 
above-mentioned methods of administration, a survey conductor should ' 
select the particular method or combination of methods which will best 
serve his purpose. He may decide that personal interviews will provide 
information on the basis of which he can outline the general areas of 
information .needed, but that the final questionnaire will be administered 
to a group. On the other hand, if respondents must refer to certain 
reco-ds, he may decide to have the forms, filled out bv the respondents 
at t.\eir individual workplaces. , 

In the investigation of candidates' qua lifi cat ions ,. whxc h is part 
of the overall examining program of the government , the examiner must 
decide whether to check candidates' qualifications by personal interviews 
or by mail vouchers (employment ref erence ■ forms ) . He makes his decision 
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on the basis of such f act: n's as the gra^'- of the job, the importance of 
variation in personaliby to successful (?rformance on the jobj and the 
number of applicants t9n3e investigated. 

Decide on the form the questionnaire will take and how the data will be 
analyzed , • X.,^ , 

Regardless of what kind of survey is carried out, the survey 
conductor should decide in ad ance the method of analysis to be employed. 

A survey conductor must extract the data from each questionnaire 
and collate uhem for all questionnaires. Then he can analyze the 
matt'-'; tl systemat ^ally. 

He must consider what data hr- 'ants to get in quantified form, and 
whethr the data will be hand- or ...ichine- tal' ulated . His decisions about 
how r.nd on wh.a basis the accumulat*.^d inf orm^: ui on will be analyzed 
influ::nce the content and item types to be included on the form as well 
as .t:hj methods to be us- d in scoring or ratini^ replies. 

The following illustration shows why planning for analysis is 
neces.sary in order that the survey results may be meaningful and useful. 
An ag 'Hcy is interested in findini^ out: how v;e 1 1- informed its employees 
are ai'out the.Lr retirement benefit :m that the* agency 'may.determine 
whether or not a ret i rement- informal , m program is necessary. The 
survr?y conductor designs a questionn i.re form which asks specific, 
factual questions abouL retirement, :d submits it either to all employees 
or to a sample (^f employees in the M..-ncy. Tht' survey conductor in this 
situation had d» :^ided in advance on "tiat basis uhe data would be analyzed 
and this insurec the inclusion of il' ins covering all the necessary factors 

Analysis of item responses showed how much correct in format! n 
employees really had and pointed up inaccuracies in their informaiion. 
The data were bn ken do\m so that the survey conductor knew wheth'^r 
knowledge was related to such factors as grade level of employees, 
types of positions held, length of service with the government, family 
status of employees 3 exposure to information on retirement benefits p-.d 
regulations^ and respondents' opinion^^ about their own knowledge of 
retirement benefits. 

Once the an.i yses were made, the agency knew whether an inf rmation 
program was nect-s- iry, whether it was needed for all employees ^ r for 
special groups ' employees, and what specific areas of iretiremf^nt inforr 
Illation should b ::phasized. 
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Cc^nsider the purp os e of the survey in terms of the above f actors 

The general purpose of the study stated at the beginning of the 
project should now be reexamined in the 1? '^t of the infon ition so far 
ii^teriTiined : what information will be gained by the study, who will 
provide the information, b.y what method it will be obtained, and how 
it will be analyzed after it is tabulated. 

'^"nis conn '^^t^ t at . [\en t of purpose is a summation of the planning 
. i ■ ..o questi.T - . 0 con- : ruction . Certain changes in the original 
gen. . il purpose m^y have t.) be made at this point if preliminar}- investi- 
gation has raised new pL.^blems. The statement or outline of purpose can 
serve as a guide t^^ planning and to admin i s t, rati on of the survey so that 
the results wilTTie co.upl-.-rte and meaningf 

\ stat'.^ment of a ^en^^ral purpose or roblem i s ' i llust rated as follows: 

Gc?neral Purpose of the 1- vject 

To develop an Insi rumen t or technique: ny which the job perf ormance 
of trainees hired from :i general aptitude examination may be evaluated. 
S Mr>tj of the trainees v. i 1 1 be employed i-. ihc main office of th^ fi rn; 
t;-r others will work in branch offices t roughout the country.. The 
ins-jrument will be used in validity studi s of this examination. 

Information Concerning Subsequent P .nning Stages 

Is a questionnaire the proper tool to us**? 

Since the general aptitude examination ill be used to fill an 
es'.imated 120 different ;nbs in approximate : ■ 50 offices throughout the 
country, validity studies will require a cri::erion rating form applicable 
to a large number of different positions. 

A graphic rating sc;i.e on which a superv/isor compares an employee 
with others in the grpap seems inappropriate for this purpose, since 
there will be difi -rences in employee groups in different jobs in 
dif£er»int offices It seems advisable to have supervisors rate employees 
on specific behavior so that ratings will be comparable from job to job. 

What information is needed? 

Supervisors' ratings of 
severa_ broad aspects of job 
develor-^d on the basis of ea: 
that t: - items distinguished 



::;nployees on items describing behavior in 
performance, 'Iliese are items which were 
ler interviews with supervisors who felt 
etween good and poor employees,'. 



To whom will, the questionnair be g^iven*: 



To supervisors (^f 
aptitude e> iinin>Lii(MK 



einn I o- 
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: lied from the gen< ."al 
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How will it be administered? 

By mail, since it will not he p> ■ 'i '.ck ' si.inervisors 

personally, 

will it be analyzed? 

Supe^r^visors ' ratings of job performai :e will be comp:.:red with 
eirpliyees' general aptitude test scores. 

i_ for continuing documentation of the process and even-gal followu p 

Accounts of What is done should be maintained with documentation 
in rder that there is written evidence of all procedures, plans, and • 
aci ons. A questionnaire conductor should ensure that careful records/ • 
ar. raaintained of each step in the process. In the first place, it will 
enc le him- to know, at any givep- stage, what has b'Ben done up to date and 
th- rationale for it. Secondly, it will be of value to oth^r persons 
w plan questionnaires along similar lines. Documentation points up 
tr. approaches that prove- successful as well as those that are abandoned. 

A third consideration is" this,: if the suryey conductor, for one 
r': oon or another", has any plans to followup on the group surveyed at a 
Ic '^r date, he must have access to all facets" of the current study -in 
'or ier to plan realistically and intelligently for th^- followup. 
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CHAPTER 3 

Defining Questionnaire Content 

The first step in the construction process is to describe in detail 
the specific questionnaire content. The list or outline of specific 
content is a detailed breakdown of th^ general areas of information 
decided upon for the study. ^ 

The constructor breaks down each general area of information 'into 
the necessary specific areas. The number of specific areas of information 
within a general area is dependent on the breadth of objectives of a' 
particular survey. 

At this stage the constructor should work closely with specialists 
in ..the particular field wi'th which he is concerned. They will help to 
subdivide the general areas into small, distinctly separate units. 

It is imperative for the constructor to keep a careful written list 
of the content details. Later, when the items are written, he 'can indicate 
on the detailed content .list the individual i tems on the fornl that provide 
information for each part of the list. This procedure will insure coverage 
of each area of required information with one or more questionnaire items. 

the following illustrations show how two different kinds of general 
information areas are subdivided into specific areas. First we will show 
how the general area of "identification" information . can be broken down. 

Identification Information 

Identification of Respondent Identification of Questionnaire 

K Identification Number 1, Title (usually supplied) 

^* ^^^^ 2. Sponsoring organization (ditto)' 

3. Address 3/ pi^ce of administration 

'^r- ; , -i. Date compl-et- . 

5. Age (birthdate) 3. Interviewer name or number) 

6. Agency or installation 

7. Branch, Section, Unit, Company 

8. Position (grade, salary, length 

of service) 

9. ■• Other: List . • . ' 
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Following is an outline of , the breakdown of one of the factdrs 
(Nature and rlxtent of Supervisory Control) that is used in determining 
standards for a "series of positions. 

1. Nature and Kxlient of Supervisory Control (for all employees) 

A. N.i ire of Assignments 

1. Extent of detailed information given in assignments with 
regard to '''^ 

a. objectives 

b . procedures 

c. methods 

d. kind and variety' of assignment 

2. Grade level determining factors of position 

3. Frequency of assignment 

a. regular 

b. cycl^ical 

c. seasonal 

d. one'- time assignment 

B. Supervisory Control over Procedures and Methods ^ 

1, Extent of responsibi 1 i ty ,f or planning procedures and methods 

a. Wio plans? 

b. VJho reviews pla.ns? 

c. Who approves plans? . 

d. Relation to kind and variety of assignment 

2, Extent o£ responsibility for carrying out assignments*^ 

Wlio carries them out? 
D, Uho reviews the work? 

c. Uho approves the work? 

d. Relation to kind and variety of assignment 

3, Frequency of supervisory control over procedures and methods 

a. regular 

b. periodic 

upon completion of work 

related to kind and variety of assignment 
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II. Nature and Extent of Supervisory Control (for supervisors only) 

A, All of the above information needed about supervisors' control 
over the work of others 

B. Exuent of Supei-visory Control as Related to 

1, number -of subordinates 

2, grade levels of subordinates 

3, specific characteristics of assignment (technical, 
inter-agency, intra-agency , rese^arch, etc.) 

^^"^--4.__-kind and variety of assignment 



CHAPTER 4 , ■ 
Selecting Appropriate It&m Types 

Decide what type of items to use in the questionnair e 

Items can be either subjective or objective^ depending on the 
method of scoring or'rating. Subjective items, i.e.\ open-^nd or free- 
answer items, are thos'e in. which the subje^ct responds to the item in his 
own words. The nature of the responses to. these items requires a sub- 
jective evaluation of the response by a rater. Often, however, he has 
rating standards to use as a guide in maKing his judgments. 

Objective items do not involve subjective judgment in their 
scoring, although the respondent may use subjective j'.^dgment in answer-, 
ing the question. Objective items are rated 'on the basis of, a_ scoring 
system which does n£>t permit raters to differ in their evaluation of 
item responses. • .y 

We cannot say that any one type of -item is the "bes^t''' type. We can 
describe the characteristics, advan tages , ' and disadvantages of each, — 
The type or types that will be most useful for a particular questionnaire 
is determined after comparing the scoring methods, methods of administra- 
. tion,' and appropri'ateness of ' different types for the subject matter of 
the questionnaire. 

C haracteristics and examples - of various item types * 
A. Free-answer, open-end items 

The basic characteristic of an open-end item. is- that it provides the 
respondent frejsdom of response. The amount of freedom ries from one 
question to another, depending on the limits inherent in theitem. , 

Example 1:., Have you any additional comments to make about the. 

on-the-job training you received? ^ . - 

Exarhfile 2: Is there anything else you would like to say' abbut . 

your job? . /^'"^ \, 

The first example gives the respondent freedom to^^^makje;' any comments 
he likes about the training he received, provided^he "limits his ^discuss- 
ion to on-the-job training. The second example^'^s a wide-open question ^ 
in which the respondent has complete freedom to discySs anything related 
to his job, , ^ / 
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In the. exploratory^ phases of a survey it is --of ten of .value to dey 
vise open-end questions in order to get ieads^. f or th.e-deve lopment of 'the 
general and specific areas of information needed; "To i 1 lus tra to > ' in 
planning a "job satisfaction" survey, it;>is"'''*neces sary to knov; the basic 
conditions'- or factors that inf luence." emp loyee satisfaction c^n the job'. ^■ 
By asking a sample of einploy'oes in a' particular office or in particular 
positions to ansv;er' some general questions designed to bring out their 
reaso.ns .f or'- job satisfaction or" disscitisfaction-, the cons true tor -ccin 
formulate the general factors in the work situation that make for such'' 
satisfaction or, dicsati sf ac t ion . Later, on the basis of the goneraT 
^factors, specific objective questionnaire items can be. prepared.,- 

Open-end questions^ ca." serve as '.'saf e ty- va 1 ve ", or "catch-all^' 
"questions. They frequently are included, to give participants the oppor- 
tunity to furnish didd i t iona 1 inf ormat i on about questionnaire subject 
matter or to discuss an area not covered in the ..structured part "of 'the 
questionnaire fprm. \^/l^en included in questipnnaires; which deal with 
■•attitudes and opinions rather than observations and facts, they may 
supply information not otherwise obtainable and also\give respondents the 
feeling that they have had a chance to write or say e»verything they wish. 

Open-epd -questions may"''be used successfully as fplloiNaap items, ask- 
ing for further detaij-s .examp les , suggestions,, or reasons, . 

Examplc-a:' If your answer to A, B, or C is "yes'|', give details in 
. . ^. item , ■ * 

' ■ ' Example 4: Wliy?- ■ . 



Example 5.: Anything else? 



Example 6: Give an examnle "6f, 



Free-answer questions that ask how many, how long, to what extent,. ^ 
etc. should be so phrased that respondents know the units, time inter-'. 
vals<, or degrees in which their answers should b.e expressed^ The- quo' 
tion, "How often do you change jobs?" would . be' improved if it wei-e/re;. 
/Worded to ask, '^'How often in^ the last 10 years '"have . you , changed jobs?"',' 
/similarly, "How much, of your time dp. you spend typ ing?'" should'be re- 
/ vised to ask, "In an average day,' what percentage of your" time is spent 
typing?" ^ - - , <' 

Tl ^- time required tcT answer open-end ...questions, is.much greate-r/^:han 
the 'time required to answer object ive^type items. This may discourage- 
.some responjients , particularly recipients of mai 1^ ques tionnaires ,' from 
completin-g-^the f orms.-a's . caref n.ll.y or completely as they, nitght. They^ ma>- 
be upwiHa^ng 'to .write'lqng answers,, or too busy to take the time to do 
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so. Furthennore, they may not feel t:hat they understand the question 
or the issue thoroughly enough to cominent on it without additional 
guidelines. 

Wlien free-response items are used in personal interviewing, they 
may increase the time required for the interview if the respondent 
gives long answers which the interviewer must record. 

Because the content and language of each respondent's answer to the 
same open-end question differ, it is difficult to compare the answers of 
different respondents. The answers, too, in addition to taking a sub- 
stantial amount of time to score because of the amount of material to 
be read, require subjective judgments on the part of the rater. He 
must try to avoid bias in rating free-answer items; sometimes a rater's 
personal characteristics, attitudes, and opinions may Influence his 
evaluation of a response even though standards or guides for rating 
the item are provided n 

It has been found that there jLs^much hi gher agreement on the 
rating of free -answer' ques tioris^ among trained than among untrained 
raters. This finding is not surprising since a training course would 
undoubtedly include discussion of the questions themselves and the 
standards of rating for each individual item/ However, ,even assuming 
trained and impartial raters, rating is complicated by differences in 
the meaning of words to the various respondents as well as to the re- 
spondents as compared with the rater. 

With some free-response questions, it is possible for the con- 
structor to decide in advance that the replies will be arrci::iged in 
logical groupings. In an analysis of data by age, for example, the 
constructor can prepare age boxes (21-25, 26-30, 31-35) etc. and ask 
the Respondent to put a checkmark in the appropriate box. 

To summarize, free-ancwer, open-end questions have definite use- 
fulness under certain conditions. However, in view of the potential 
difficulties involved in their use, it is well, in general, to consider 
using^t:|ier item types when they can appropriately provide the necessary 
information. 

B. Short-answer items 

Short-answer items are a limited kind of f rn^ n n-^t.^nr item. They 
are statements or. questions which require tht to supply a 

brief answer in a space provided for that pu 

The respondent is given more leeway in his answc ^han in forms 
where alternative responses, arc listed. The item t. ^ ic and- wording . 
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determine whether iiidividual short-answer items assume the characteris- 
tics of Tree-answer items or of objective items. The more freedom the 
subject is given, the greater is the resemblance to open-end items and 
the more difficult the objective scoring. 

Example 7: (limited) The lowest salary 1 will accept is $ 

a year. 

Example 8: (fairly unlimited) -List che comparable jobs to which 
these emp loyees could be transferred . 

Short-answer items are used in questionnaires when the topic of the 
item is such that the number of possible responses does not permit the 
.item to be written in mu 1 tip le -choice form but the constructor, never- 
theless, wants >o put as much limitation on the item as possible. 

Completion of short-answer items sometimes depends on the individ- 
ual's recall of information or on his supplying information obtained 
from records or other sources . 

Example 9: 

Fill in the columns below for each of your arrests for violating a 
'driving law (not parking) DURING THE PAST TEN YEARS. Be accurate and 
complete. A check of police traffic records and court records is made 
routinely for every tappli cant. 



Arrested for what? 
(speeding, driving 
while drunk, pass-, 
ing a red light, 
etc.) 


Wliat * ' 
year? 


Wliere? 
> (city 
or 

town 
and 
• State) 


Were 
you 
con- 
vic- 
ted? 


Was permit 
revoked or 
suspended? 


How much w^is 
• the court 
fine or the 
sen tence ? 
How much was 
the fine or 
the collat- 
eral without 
going to 
court? 
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K:%;inip U; lu: TIa* iiuinbor t)£ reports of rating niailed by your 
soc:tion last month wa.s 



in Lhi.s lm.sCj the rct^pondcnl: may not rcunoinbor the iiumber of reports 
inaLIed by his section during the past month, but ho can Tind out and re- 
port the proper number on the Jlornu 

I'lAainpIes 9 and 10 wjuld not be satisfactory for iiiclusiun on a 
ques t i onnai rL» f.orin to be administered in a group situation where records 
or other sources of infcirmation are not available. F.xamp le 9 is taken. 
{ rom a form which an applicant completes at home and submits wlien he 
applies for a position, Kxamplc 10 could be used successfully either in 
a perS(.)nal interview or on a form filled out by the respondent at his 
d e s k , 

C , D i c h o t o 01 o u s or t wo - r e s p o n s e items 

The dichotomous item form provides the respondent with only two 
a L t ernat ives from which to select his answer. The alternatives consist 
of Separate, mutually exclusive categories^ such as "Yes-No", "Agree- 
Disagrce" , and "Righ t- Wrong , " 

i^Kaiui^le 11: Are you now employed by the Federal Government;? 

Yes 1 

N(; 2 

Tv;(^-v;av/ questions are at the opp os i te ^ ext reme from free- answer cjuep- 
tions. Open-end items give the respondent freedom of response, but di- 
chv.)tomous items fc^rce him t(^ decide between two choices. Forcing a de- 
risicMi is satisfactory provick.d that the issue ^involved is a clear-cut 
dichotomy and the choices ai'e realistic, I 

Many issues have only two sides, e.g., "is or is not" or "know or 
kn<^v; not." Sucli issues are particularly appropriate for dichotomous item 
construction, 

F.xamp le 12: Are you currently engaged in more than one paid 
employment? 

Yes Qj No CZi • 

Example 13: Check one: Hale . | | Female [ ^ 

Dichotomous ques t ions are common ly used in comb inat ion with other 
item types .when the basic issue uestionably 't'.^o-choice , but further 
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detia ^ 
pe L's 
This 
tilt* 
cho i 



:i7*^' roc [ui red. 
reccivt ,:: 
De asked . 
, tliiality, LiK 
les L ions , 



t^Ic, it 
i trainii 
ice cjuusi 
Lx-aii^ . 



rable 
L. ion V. 
t:her c 
Laiu' 



> find 
Lb his 
tails vf 
from 11! 



)-wa\' ciuesUi 
has -Lear undcrsi .ncl 

ample lA: I c.r . 

Yes I 



♦ • carefull wore. su that Lhe resp. 
wl 1 at info rw ; a t i. o n is wa n ted. 



iio cafetc r 
No □ 



II uv ')uildin^: very da> 



If a respondent c\i c 
lunch from hon^e ever;.' (h; 
that he buys as well as 
brings part of his lunch 



r "Yes", it. -CMuld m that h- brings his 

iid eats it while .. iiting in the cafeteria, i 
ltilS his lunch there. It could mean that he 
;iiid buys the rest in the cafeteria, A "No", an- 



swer could indicate that he docs npj^e of the possibilities mentioned, 
i^ut eats outside the biii Icing most or all of the time. 

The wording and m-^aning in this example are ambiguous all the way 
through. The word "eat" i .^; not defined clearly (is it buying or actually 
consuming ttie food?) and the expression "every day" is so limited that 
many people who "eat" in the cafeteria almost every day would hesitate 
to answer affirmatively. It- would be difficult for the respondent to 
know how to answer because he would not know precisely what information 
was needed. The rater would not know what, the responses really meant 
siiHce so many interpretations of the item are possible. 

1 In two-way c[uestions, both alternatives are not necessarily stated. 
If -i we ask the question, "Arc you going to the library today", the im- 
plied negatrve "or not?" seems . so obvious, and the possibility of mis- 
interpretation so slight, that^ it need not be included. Studies have 
shown, however, that no matter how clear such questions appear, it is 
posisible that the assumption may be challenged. In case of any -doubt., 
it iis always safer to state the question completely. 

i Many public opinion surveyors f ind that dichotomous questions are 
advantageous because they are easy to answer and to score, and can be 
designed to fit many situation^. They feel, too, that such questions 
comi3 closest to dup 1 icat ing the types of decisions which most people are 
acci|jstomed to making. This does not mean that the final choice of re- 
spor|vsG to a two-sided opinion or attitude question is simple. Before se- 
lecting his answer a respondent may have made several decisions of which 
he is unaware. V 



One of the limitations of even carefully written dichotomous 
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opinion L^r attitude ( lesti 
a general or overall ^pini^ . 
tion of reasons for t te opii: 
fications the respond nt may 

As an example, one might , 
or not liking a job, A respc 
dislike certain aspects of it. 
supervisor. He may like his 
would not make a change unles 
the other hand, he may be hap 
special offer were made him, 

i ■ the respondent were a. 
his answer would not yield th 
vey conductor might find it us 
question such as this one, and 
components of liking or not li 
put in multiple-choice form in 
about specific factors such a.^ 
conditions, and salary. The r-Lt.;; 
response to the original "like' 
data for analysis of the elemc:... 

Issues concerned with attlt:-^. 
yet respondents are frequently :i. 
questions. Such issues frequer . 
only two choices are given. 

Example 15: Have layoffs ••e:-. 



G final resp ;ise indi::. only 
It does rut give ^iii}' dica- 
intensity of it, or 'i.' quali- 
'It, 

:\e questiion relating to - king 
enerali;- like his job, buu still 
ike his duties but not ' s 
king conditions so well Iiat he 
i:d inary opportune ty art . On 
^ut willing to change i a 



ou or do you not like your job?" 
information. However, 'he sur- 
. .;ciude a dichotomous generalized 

lude questions to investigate the 
I'b, The latter questions could be 
.:o obtain the respondent's opinion 
vision, trainings job duties, work 
to these items, together with the 
)t like" item would provide valuable 
. malve for' job satisfaction. 

md opinion are not always clear-cut, 
to express the In dichotomous -'type 
oduce a third response, even though 



Led faii.-ly? Yes 



No 



The extra response may be .^s", "do::'t know", "both", 

"neither", or "no opinion." If j -;uestion is ..presented orally, the 
interviewer records the unsolic Lii_rd response. Op a written form, 

however, some provision must be made for the eKtra response; otherwise, 
the question may be omitted. "nere would be no way of knowing the reason 
for the omission and the ques on would not yield any information. 



The writing of dichotomous 
opinion (like the writing of ar 
nical job. The public opinion 
changes of a word or two m a d. 
response to that item. Unless 
ist in'' the investigation of at: 
to get technical assistance. 



ie _ " i as to inves tigate attitude and 
att.Lude and 'opinion items) is a tech- 
tt, _zure is filled with examples of how 
•hot rmous it em significantly change the 
que ticnnaire constructor is a special- 
id e- and op in ion , it wou Id be advisable 
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D, Multiple-choice items 



Mul tip le - choicj ite.is are composed of an introductory part, call 
the item premise, and a list of three or more alternative responses 
which a respo::dent -ele.. :s at least one answer. His freedom of choic 
greater than in a l ;o-c.. ace ciuustion, but. is more limited than in 
response item, 

N 

Multiple-choice items, have the following advantages: 

^Jhey are easy foj the respondent to complete. 

The ro^,j^ond^><fE does not have to express in his own words the 
formation desired, as he* does in an open-end question, Alth- 
he has more material to read, he does not have to write or g 
oially an answer which may be lengthy « 

Recording the response is easy because the respondent merely 
has to make some sort of mark,- i.e., place a checkmark next 
to a response or in an answer box, circle the proper answer, 
or blacken a space on an answer sheet. 



2. They are' easy to score or rate. 

Scoring or rating is also a simple process; items can be scored 
by hand or by scoring machines. The responses can ..be lettered 
or numbered and arranged so that the coding and extraction of 
data are routine clerical operations, Mu It ip le -cho ice re- 
sponses, like those of other objective item types, can be pre-* 
coded. In checking the response, the subject is .actually 
coding or classifying his answer. 

3. They can elicit varieties as'Well as degrees of iriformation, 
opinion, and fact. 

A t>;o-way question provides onl-y responses which represent a 
ch^ ice between two extremes. In some cases, e.g., the male- 
female dichotomy, the number of responses is obviously limited. 
Or the other hand, open-end questions can. produce such wide 
varieties or so- many gradations of opinion or factual informa- 
tion that categorization or classification is difficult or 
unwi.eldy. For example, place of birth (city) permits a practi- 
cally limitless number, of categories . Multiple-choice items 
are inot suitable in either of the above illustrations ; they are 
appropriate when the minimum number of responses" is three, and 
the maximum tiumber is finite. . 
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7ier\* ( response 



T.plu- n. ( twc 1 Lee iarit . is: 

Mar- J Singl- I [ 

(multiple- :.arit sz-.^-^s: 
choice a. "^rr C^-d "T 

b. ivcr-::ad 

c. -epai.ited 

d. idcved 

b. iisuvvniont o£ degree 

Eiicimpiv 17: (two-choice) What is your opinion of the 

employee ' s overall compe- 
tence? 

Satisfactory I [ 



(multiple- 
choice) 



Unsatisfactory 



Check zhe box that repre- 
sents '"ou:r opinion of the 
'femploy^e's vera 11 compe- 
tence. 



! 1 



Multiple- chc ice 
allow the wite: 



Outstanding ' 
Good 

.Fair 'i I - 

Poor I . I 

:.-ms may t>e written in var , .^us forms that 
lexibilit .n item construction. 



The iteir. cc- '3truct:or sele::- J'le articular nultiple-chcice 

form whizh most suita: or e h item tcric. A single 

ques tionnai ':e rrav contain era. 'Afferent riinds of nultipie- 
choice iter.3. 

a. Fl~...: -quiring only one inrwer 

1 Inc mplete premise and three cr nore alternati es 
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1.U 18: In hie ■ • I t:-.>nl: a 

a) coi-eg'! icepara-ory course 

b) - 111:1"-. irts co. rse 

c) co;-;:'.-- .il arts -iurse 
' ■ dT me. al arts '.lourse 

i) cor./ ther th..:i those lis'zed 
ar - 

re premise (dir^_. : question or statement and 
or more alterna es) 

le 10; V/liicl: o: l following best repre- 

senu:: y -n.ucationa: level as of the 
end of tv-. .school year? 



Some high ..chool L 

Completed dgh school L 

Some college or specialized 
school a'^iove high school 
level ( uch as business 
college or nursing) 



Bachelor 3 degree or equi-^ 
valent 

Som : raduate work^.. 

'La' .:ree 

-: iegree or equivalent 

Jc: " /-^ above the master's 



\ i 



'I i 

[ s 



3c -^egr^e or equivalent j ' 

re ;ui~:irz; "^z r -ha: zne answer 
-:ccnipl^,r^ pre:-.\ie a:^ three or more alternatives 

xample .J: Fi-^ase cT?^,ck the appropriate space or 
; spaces . 
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I have l;nc)wr 
I \ an unde 

I ! idi. 
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, o t " r 



andidate as 

.'dent 
i s tan t 
" J is tant 

. ') 



uipi-ete premise and thre- r more al tern .itives 

.-.iiple '^/Lieck all o: : ^^e. following p )Sition 

evaluat on ■ ods whic' y , use to 
determi ' U s of wor--., 

Fed-£ral ' sition Cl£-: : if icat i . r 

■J jD Ranl:^ nt> 



J 



^ an 



I i F."Ctor C:':ioarison 



Me: >d (rating t le 
iiCtor ciigree values 

r-- (de.-cri'be) 



c ♦ Forms to determine degr .imc j.:it, or intent .2/ 

In ques lionnaires des: r - ■ : J mer'^-re the opinions* n- 
" teres ts, :nov;ledgr :r ' :Les the respondents _t 
is ofte'- :i. a- :.'---r7ine ene degree, amount, r in- 

tensity. 7ar a^ r-iati s .n ach question are deg- es 
or stepF in .l. c: nuuv. Tae -^ange of the aontinuur is 
from one t^i^itrenic am iaer (most to least, like to is- 
like e;:cellent 1 )cc - . The steps are graduated, in.e 
number cf 3te::s de ancirg or. tae degree of discrimin ion 
desired. The iteni cons ".:rucL:or makes the limits of tr. 
continuum as dde or as narrov; as is necessary or app' O- 
priate for th - topic. 
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Example 22: Hov; . . you feel about, ti. 
sales t/.xV 



approve 



J Stron, 

J Appro- - 

J Undec ied 

J Strongly disapprove 



posed city 



As in the example abz' it is often necessary to provide 
a response such as '■"j:rc:-icided'"'5 "have nc definite upmion'S 
"not sure" 5 etc . In r;!,- ■ que s t ions , hov. . /er , px^oviding 
such a choice permits res;: Tidents to avr j giving' Jei 
ite answer when sucli 



Example 23: What do y j-u 
supervisor? 



v/cr is :^os s i\ , 
--ink of your unmediate 



a. . L_^ve 


very muc 








b. Like h 


f - - y w 








Dislik- , 


i- : mev 


■-.it. 






i. Dislik- 


- -rry 


rue: , . 






3 . Don ' t kr 










Since everyone knows vhe' 






: kes his sur 




visor, a noncommittal C- 


r sue. 




- *t know" ,.r. 




/..j\ot be inc luded among t:_ 


^rerna 








Ranking items are used to deterrai*. 


relativ 


^ s ran 


a^gs • The 


re - 


spondent is asked to arrange a list of sz 


:irement s 


, w: rd 


phrases. 


or 


people in the order of his preference. r 


^ ranks 


sue', a 


-.ist on tk-:^ 


b«i:- IS 


of a given standard. For example, emplc 


:es may 


bo asl< 


-a to rank e 


ack of 


several work duties on the basis of "-h^ r 


. lative 


■amour • 


V . time sp€ 


ni in 


their performance. \\he stan 'd i'\ t_;.,' 


case ir 


: ht- s 


e lative amr. a 


nt c • 


t ime a n emp 1 oy e e s p e nd s per'* -ni:. - e e m c r 


f r -ese 


r ■ 1 cr s 







Although mos t ra-^ ki 
of the reSi>ondent, th, 
tained . 



^'olv' ;jb jec". Ar. " j- .a ' 
N-e i;/::th tc analyze 



th 



tj a r 



A constructor^ must be awar- of li serious _li:ritat. n of ranki-^r 
items. He. cannot assume that the 'rvals between th ranks are 
On ap item in which the respondents raak job duties ac. rding to the 
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amount of tir 
t ime- cons urn i j, 
the duty to 
p lace 3 howe\ r 
spondent ass io 
ings of all re 
average (moaTi) 
, ing of each i 
the duty which 
to perform as 
, certain duties 



spent on them, a job- duty given the rank of "1", the most 

duty, may not necessarily take very much more time than 
ich a rank- of "2" is assigned. The duty ranked in third 
, may take much more time than the duty to which the re- 
ns fourth position. By statistically .combining the rank- 
spondents to a particular item, a survey conductor can gc^ 

ranks ^ tor each factor and measures of spread, in the rank- 
L' tor among respondents. He can not conclude, however that 

rec;eive(d\n average rank of "3" took twice as much time 
duty w:th an average rank of "6", He can conclude that 

took more or less time than others. 



Exarr.pl (f 1'4 ; 



Listed below are various, job duties . please rank 
these duties according to the pa'rcentage of time you 
spend On them in an ayerage week. ^Write a (1) in the, 
right-hand column opposite the duty on. which you spend 
the greatest percentage of your time, a (2) opposite 
,the duty on which you spend the next largest percent- 
age of your time, etc. If you spend a substantial 
'amount of time on a duty which is not listed, indi- 
cate- both the duty and its rank order under "other 
(specify) 



Job Dutv 



Assigning jobs 
Obtaining equipment and 

materials 
Reviewing employees' work 
Training employees 
Keeping records 
Scheduling work 
Devis ing improvements 
Attending meetings 
. Other (specify)__ 



Rank Or der of Job Duty - 
bv _Amount of Time Spent 



Descrip t ive items involve the rating of people or variables without 
comparison in ranking items) to other persons or variables in the 

same groL • Items can be use- to gain information about one or 

more pers. : .ciab.les with respect to certain opinions, attitudes, 

interests, • The item categories or steps on the continuum represent ■ 
a systemati2ation of judgments, assessments, or ratings from which the 
respondent selects his answer. 

The degrees or steps in the continuum may be defined in brief, ab- 
stract terms, or in more ' de tai led descriptive terms. In either case, the 
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constructor should be certain tha^t the alternatives are. separate and 
clear, and that they cover the range of the particular interest or atti- 
tude completely. 



The variety of descriptive item forms is limitless, \^[^en construct 
ing an item, the constructor can adapt an existing form or he can devise 
a form suitable for, the material with which' he is working. The forms 
differ in format and wording, but all require merely a checkmark, number 
or symbol to complete 'the items. 



ExampLe. 25; Does your supervisor give you clear, detailed, 
complete directions about your. duties? 



and 



always 

most of the time 

some t imes 

never 

Example 26; Does your supervisor give you adequate instructions 
about your duties'? 

His instructions are always clear and complete. 

His instructions are usually clear and complete, 

but I occasionally have to ask him atout one ■ 

or two details. 
His instructions are often unclear and incom- • 

plete, and I frequently have , to check details 

wi th him, 

Hi s ■ ins tructions are rare.ly clear and complete. 

Example 27; For each task, circle 

(L) if you like todo it 

(I) if you are indifferent*, and don't care whether 

you do it or not 
(D) if; you dislike doing i;t. 

1. Replace fluorescent bulbs. LID 

2. Install an air-conditioner L I D ■ 

5. In multiple-choice items, the constructor does not have to 

depend on the respondent's recall of al/. possible alternatives 
before selecting a final response, as is necessary in free- 
answer items, because practically all of the possible answers 
^re provided, -Even though the number of possible alternatives 
in an item may be great, it is still preferable to write ..the 
item in an objective multiple-choice form than it is to sift 
through much written material, -Preliminary research usually 
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brings out almost all of the expected responses, and the small 
proportion that are not provided for can be written in an "Other 
(specify)" category. 

Example 28: (free answer) List all- technical manual's which you 

refer to in the course of your regu- 
lar duties. 



Example 29: (multiple-choice) 

Check all technical manuals which 
you refer to in the course of your 
regular duties : 

TM 201 * ' . 

TM 203 

TM 204 

TM 208 

TM 219 

' Other (Specify) ^ 

When completing the free-answer question, an employee might forget 
to^ list one or two of the manuals, particularly if he does not use 
them frequently. In the multiple-choice question, the. constructor 
has included all the manuals which, by investigation, he is reason- 
ably sure that employees in the particular job normally use in 
their work. A respondent is unlikely to omit any since the list 
serves as a reminder to, him. 
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CHAPTE. 5 
Classifying Questionnaire Data- 



Decide how ^ code and e x ^ ^ : d a t: a . ^ 

Earlier in this pamphl .^e Lis ,ed practicality of scoring as one 

of the essenuial characteris : s cf a good .questionnaire. Planning for 
scoring and extraction of d?:a is part of- the process of constructing a - 

questionnaire, although it i not a separate or exclusive step in the 

process. It is a continoin. ::onsideration during the stages of deciding, 

on the types of items to be ncludec , their arrangement, and the layout" 
of the questionnaire* 

How the data will be a: -Vzed should be decided early in the question- 
naire construction :i±nce th will affect the types of items used, their 
scoring, and possibly their arrangement in the questionnaire. 

In order to extract caL_ from completed questionnaires in^,^ Qrderly 
and accurate way, a coding (scoring or rat ing). system is devised tocover 
each item on the questirnr^aire fcrm. A system for codirrg the responses 
consists of assigning numbers or symbols ( le t ters ,^^arbDre ^iations , e t c , ) 
to each possible answer. Thi: codes classify the 'responses ' so that the 
information received from respondents can be^easily extracted and tabu- 
lated preparatory to ana ysis. The respondent may select his answers in 
accordance with the prescribed code , specif ied in the directions given 

him, or the interviewe r may code the respondent's answers, In' cases 

where a predetermined code has not been developed,^ the coding is done 
by trained: coders after >he questionnaires have been completed. 

The type of -c'bding system used "in questionnaires is , in general , 
..dependent on whether the data are to be tabulated by machine or ,by hand. 
Machine tabulation requires- numerical or alphabetical codes. For hand 
tabulation, any coding syn:bols may be used. 

Exceptions to usual coding procedures are the methods used for 
rating application forms and .qualifications questionnaires (vouchers) 
as part of the Federal examining process. These rating methods are 
described in appropria~.e government publications. 

Coding of different ::ypes of items presents different problems which 
should be kept in mind waile individual items are being designed. / ' 

The development ot a coding system for items is a complicated process 
requiring the "preparation of groupings and subgroupings of possible yf^esponses 
to which group and subgroup codes.xan be" as signed . These groupings are 
made'^n th-.^ basis of the results of pretesting the items. / .-""'^ 
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Open-end items, which are scored subjectively by a rater, are 
difficult to code. The rater reads each response and assigns what he 
thinks is the appropriate code to it. ^ . 

Since the rater must, in .nany cases, use personal judgment in the 
assignment of codes , the reliability of the coded ratings from one rater 
to another, or even for the same rater on a later reratihg, may not be 
statistically high enough to be acceptable, 

* 

There are, of course, times when a questionnaire author feels that, 
for his purpot^fe, free-answer items would be the most suitable form to use, 
in spite of the difficulty of rating them. However, he should be sure 
that the information neeoed cannot be obtained from a different item 
type that could be handled more easily. 

Many objective- type questionnaires contain one or two free- answer 
items. The rest of the quostionnaires can be codo.d routinely, and tlie 
open-end responses can be rated carefully in accordance with a pre- 
arranged coding system. 

The coding of ob j ec t ive - type items is ^ simple and routine matter. 
Two-choice and mul tip le- choiCie questions can be set up so that the coding 
"is actually done by the respondent at the S£:-e time that h^^ chocks his 
response. 

Example 30; Are you a citizen of the United States of America? 

Yes 1 
No 2 
X 

.'(Circle the number to the right of the antswer vou select,) 

The numbers "1" and "2" are code numbers and the "x" is also a code 
which the questionnaire reviewers use to indicate that no data are obtained; 
that is, when the question is omitted or both alternatives are circled. 

In an item such as Example 30, the proper number (which is the same 
as the code symbol) is marked either on the questionnaire form or on an 
answer sheet by the respondent. The extraction" of coded data can be done 
directly from the answer sheet or questionnaire form. 

There are two main advantages in using a code system which, involves 
the numbering or lettering of item responses (precoding). First, later 
coding of reponses is avoidea, resulting in economy of time and money. 
Second, the possibility of error in marking the codes by a second person 
is eliminated. 
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Examples like the one above normally appear on questionnaires 
completed by the respondents themselves. ' Similarly, in recording 
subjects' responses on an interview form, an interviewer marks codes. 

Example 31: Respondent's age: | | 20-29 

r~| 30-39 

I I 40-49 - 

I I 50 or above 

'In the above illustration, it would not be necessary for che inter- 
viewer to write down the respondent s exact age; he would merely check 
the appropriate box. Thus the question is answered and coded in the 
same operation. 

One possible difficulty may occur in preceded items if a large 
percentage of responses to an item are found in a "catchall** or *'Other 
(specify)** category. 

Example 32: Responsibility for this work is assigned to which of 
the following organizational levels? 

I I bureau 

I I division 

□ unit 

I I other (specify) 



If many responses fall into the last category, a reexamination of 
the alternatives must be made, and the categories made more realistic. 
Such a difficulty is less likely to arise if questionnaire items are 
pretested, 

/ 

/ Short-ansver items cannot be precoded because the respondent writes 
his response, or the interviewer writes the respondent's own words. How- 
ever, the questionnaire constructor can, in advance, design a coding 
system for short .nswer questions to which he feels only a limited ^number 
of alternative answers can be given. Codes would be marked after the 
form was completed by the respondents. For example* a short-answer item 
might ask the subject to fill in his annual salary. Later coding of 
answers would be routine because codes could be assigned to different 
salaries on the basis of predetermined categories ( $3000-$3999 ; $4000- 
$4999; $5000^$S999. etc.). 

42 
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■CHAPTER 6 
Writing the Questionnaire Items 

Item writing is essentially a creative task. It is therefore diffi- 
cult to provide a writer with a list of directions that will enable him 
to produce good items every time. 

Some of the general principles which apply -to the writing and improve- 
ment of the questionnaire items are discussed in this pamphlet. Success 
will depend on how carefuU)' the writer applies these principles • in his 
writing. "-^ ; 

Before attempting actual item writing, the constructor must familiarize 
himself completely with the purpose of the questionnaire and its specific 
content. He must know how the form is to be administered and be familiar 
with the characteristics ^of the respondent group. Knowledge of the types 
of items available for use, as well as their advantages and disadvantages, 
and the possible methods of classifying the information are also essential. 

When a writer develops 'a questionnaire *with the help of subject-matter 
specialists, he and the specialists should work out the first draft of the 
items together so that errors in content and in form and wording are 
prevented. 

If possible, the constructor should devise more than one item dealing 
with a. specif ic subject-matter topic even though only a single item covering 
that topic will be included in the final questionnaire form. Each proposed 
item should be arranged oj: worded differently, although each should be as 
well constructed as possible. Sortie of the items will prove useful and 
others will be eliminated on -the basis of reviewers' , comment s\and prelim- 
inary and final tryout. In many studies, two forms are designed for 
later comparison. Construction of more, than the minimum number of items 
in the beginning" wi 11 facilitate compilation of the two forms. 

The constructor must maintain a careful record of each item. This 
record will facilitate the sorting of items by content to insure adequate 
sub ject-m? : 1' ; coverage, and aid in the compilation of the questionnaire 
items, ' .-eviews of the items should be attached to the original item. 

Each item should be neatly written or typed on a separate card or 
sheet of paper which should also include the following information: 

1) the specific area of information which the item is designed 
to cover 

2) the sources, if any, (bocks, pamphlets, interviews, records, 
etc.) from which the item or basis for the item i6. taken 



43 



39 



3) the writer's name or initials 

An individual questionnair** i'.A'v.) must meet the same requireincmts 
that any other testing instrumont must meet in order to be most effective, 

A good questionnaire item; 

A) is limited to a single topic; 

B) is appropriate to the subject group and administrative method; 

C) can be scored or rated in . accordance with a predetermined 
sys tern; 

D) is written in precise, understandable, unbiased language; 

FJ is arranged in a logical position witlin the questionnaire; and 

F) yields valid and reliable inf ui^inat ion . . 

Although an item cannot be satisfactory unless it meets all these 
conditions, the conditions are interdependent. An item will probably . not 
be valid and reliable unless the other conditions are fulfilled. For ex- 
ample, one which contains language that -45 too technical for the subject 
group is not likely to yield either valid <^r reliable information. 
Similarly, an item which is inappropriate for the particular administrative 
method cannot be valid for the purpose for which it was designed. The 
characteristics of a good ciuestionnaire . i tern, are described in further 
detail as follows: 

A. The subject- matter "of the item must be limited . 

The subject matter of each item should cover only one specific topic 
with the topics taken from the areas of information previously decided 
upon as essential to the study. 

When more than one idea or topic is presented in an item, the respon-- 
dent has difficulty providing an answer or oelecting a response. Such an 
item is called "double-barreled," Analysis of responses will be meaning- 
less because there will be no way of knowing to whic.li idea the subjects 
really responded. 

Example 33: Are you able to discuss work problems with your 
supervisor in private and in confidence? 

The example is doub le,-barre 1 ed because it is actually asking tv. ^ 
ques t i(ms- -can you discuss work problems with your supervisor in private,* 
and can you discuss them with him in confidence. 
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Tha t Lhc topic must be re levan t to , and valid for, the purpose of 
the survey appears to be an obvious point. However, it is mentioned ^ 
here as a reminder to a writer to check the subject matter of each item 
he prepares against the list of specific information needed. It is poor 
practice to ask for unne.cessary inf oriT.ation . 

B , Vne i t_em must be appropriate to the subject firoup and administ rative 
condi tions . 

A constructor must familiarize himself in advance with the character- 
is tics, of the group to whom a questionnaire form will be submitted in order 
to insure that the ' con ten t and language of the i terns are adapted to the 
particular group. He must be certain that the group members are likely 
to provide, on the basis of 'their experience, knowledge, or observation, 
the desired information. In order to investigate the relative effectiveness 
of different aspects of a recruitment program, for example, job applicants 
are asked from what source they learned about the par ti cu 1 ar examina ti on 
for which they applied. The question should be worded so that, the 
individual applicant will report only on how lie learned about, the examina- 
tion, not on how his friends learned about it. 

The language of the item must be appropriate for the respondent group 
in order to gain reliable information. Since item wording will be discussed 
in detail later in this section, only one of the considerations in item 
wording, its appropriateness, will be discussed here. 

The language of the item should be such, that the best educated of a 
group do not f eel that the language is beneath them, yet it should be simple 
enough for the least well educated to understand. When the sample group 
is either of mixed educational 'background or of f ai r ly low-leve I education., 
the words used should be comprehensible' to "'the least well educated of the 
group.- When the^ Subjects have a high level of education, words of greater ". 
difficulty may be used than would be suitable for either a mixed or low- 
level educational group. _ The more syllables in a word, the more difficult 
it is, and the longer the sentence, the more difficult the material is to 
comprehend. There is some evidence that material is m(^re readily under- 
stood when it is read than when it is heard. Therefore, for paper-and- 
pencil forms, the language of the items may be more difficult than could 
be used in personal interviewing. 

Special care must be taken to avoid including technical or trade terms* 
in an item unless it is certain that the respondents will clearly under- 
stand them. UT^ien it is necessary to use such terms, and if there are any 
doubts abcuit their being understot^d, they should be explained. An explana- 
tion would thus insure each respondent's having the same orientation before 
completing the item. The constructor might, however, determine by item 
tryout how well the 'terms are understood. 



How does one decide whether an iLem is appropriate foi* the administra- 
tive method which will be used in a study? The respondent must be able 
to complete the item under the specific administrative method, unless the 
item is one which may call for a **no opinion" or "don't know" response 
under any kind of administration. An item on a. form to be completed in 
a group situation cannot ask for information for which the , respondent 
must refer to records, but such an item could be used on a form filied 
out at the respondent's usual place of work or by personal interview. 

We assume that the ' appropriateness of item content for the particular 
admini s trat i ve method has been cons id e red in the course of planning for. 
questionnaire construction. Nevertheless, it is advisable to keep this 
in mind so that an unsuitable item will be e 1 iminated ear ly in the con- 
struction process, 

C . The item must be callable of bein^ scored or rated in accordance w ith 
a predetermined system , 

' To fulfill this condition, the format of the item,* that is, the yp^ 
of item and the provision for response, must be in accordance with tl .. 
scori*^ or rating systc planned for the form, 

A form to be s . '^r _ objectively must be composed- of individual 
which can be scored o: -ctively either by hand or by machine. In writ ■ 
the items, the const uctor uses the scoring, system as a guide to item 
type and directions for response. The scoring of the whole form will be 
easier if . as many items as possible have the same format, although it is 
not wise to force items into a particular type merely for this reason. 

Methods of response which call for placing a 'check (✓O or a cross (X) 
next to the proper answer or in an answ,er box, crossing out all answers 
which do not apply, circling the proper response, writing in a symbol, or 
fil.ling in a space on an answer sheet are all suitable for objective 
scoring. 

Open-end and short-answer questions require only that sufficient room ^ 
for the responses be allowed on the form. Scoring of such items, however 
can be facilitated if the ques t ions " are worded carefully and limits sef 
on what is asked. If short-answer questions are written so that only one 
or two words or numbers are needed to answer them, they can be scored with 
comparative ease, ^ 

Another way to facilitate the scoring of open-end questions is to 
break one broad question into two precise and limited questions so that 
the responses are easier to classify. Still another w^y to simplify the 
scoring process is to provide the respondent with fairly detailed directions' 
for completion of the. item with the expectation that he will thu-s limit 
his answer to pertinent information. The rater will, of course, then 
have less\ extraneous material to read and discard, ■ 
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D. Precise.y'^understiandable, unbiased lan^uafte is essentzial . 

.X ■ .Specialists in the measurement of achievement, attitudes, and opinions 
have done much research on the effects of wording in their efforts to 
^"'develop precise measuring instruments', . • ^ . ■ 

Some years ago, a sample of researchers was asked to list what they 
considered the chief defects of commercial research. The criticisms that 
they mentioned most frequently and probably still hold true today are as 
follows : ^ . . 

Improperly 'worded questionnaires 757q • 

Faulty interpretations 58% 

"Inadequacy of samples 52% . 

Improp- - sl.r.tistical met nods 44% 

PrQ^en: ition of results with L • - 

supp 'tin: data '^l^ 

There . - definite principle ihat must gove'rn the ch' ice of item 
language, . ^me of them may appe.ii: simple, yet. the writer who is surf- 
that he knc them must remember t:hat the important thing is to know 
them and' f( : low . them. 

The principles are not independent. One may be an ampl; xcationof 
another; one may even appear to be a contradiction of anotli r. Not all 
the principles apply in all situations, . It is tlie writer's respon'sibi 1 i ty 
to guide himself in item construction by the principles which are most ' 
appropriate to the particular situation in which he is working,. If he 
is unsure of the wording of an item, he may write it '-ith alternative 
wordings and await the results of review and tryout before deciding 
which to use, ' • 

1, Use precise language. 

Precis^ language is lanuuage that is exact, that means exactly 
w!~. rX the writer intends it i - mean, P re c i s e ne s s of 1 an guag e in e an s , 
too, that all superfluous w^ .is in an item are eliminated, Extra 
words which do not serve to ..u.'fine the particular problem may ' 
detract the attention of the subject from the main pr(^blem. . - 

2, Use simple language whenever possible. . " v 

Simple v;ords ^are preferable , provided they express the idea 
clearly. Simple words, however, should be examined.- careful ly to be 
sure they do not have more than one meaning and that they are 
sufficiently precise. For excimple, a word like "office" can be 
ambiguous'. Given a question such as, "\Vhat is the total number 
'of employees in your office?", it is not clear whether the writer • 
"means the office (room) in which the employee works- or the office 



47 



43 



(organizational di vi si on - -uni t , section, etc.) in which he is employed 



To illustrate further, 
local government. 



'government" can mean Federal, State, or 



3. Keep the. use of technical la^nguage to a niininium. 



Even when 
the . spondt 
Cer-J:in w Td 
As an ill is t :■:- 
submits in ' rci 
Vouchers ar a 
the Fede al se 
audits. 't. er 
classifier-: 



a constructor 
gre jp , he shot: 
ised v; : thin tht- 
lion, there are 
:^r to be reimbu 
I so mail qualif 
rvice perform i 
employees perf 
or personnel i 



ls familiar with' the characteristics of 
Id e: .mine each technical word he uses . 
Fed .:."-l government have special meanings 
tra :1 vouchers which a Federal employee 
rsec for ^ ^^oved t rave 1 expenses . 
ica: -^ns ti onnaires . ..Auditors in 

ndu5 rial CL..,t audits and internal 
orm iisk audi ts as part of a job 
nspe rion program . 



The vds used should expre.'^:.> the exact meaning that is Intended, 
and the sub't^c;:s must know that m-^aning. It is preferable tc 
lengthen an item by an explanation of terms rather than simply assume 
that all respondents are familiar with them. 

The item constructor should be cautious about the use of abbrevi- 
ations; one which is well known to the writer may be unfamiliar to 
some, respondents. However, i' an abbrevia ti on - 1 s explained in the 
first item. in which it appears, it can be used in succeeding items 
without explanation. 

4. Specify al 1 s tandards f or judgment, qual 1 f l.cati ons , or limits. 

A writer should Include in the item or in the directions for its 
completion all information ..necessary so that all respondents wilL 
answer It on the same basis. 



Example 34: 



IVhich of the following statements regarding your 
employment status Is true? 

a. employed full-time (30 hours or more a week) . 

■ „ ^ . ■ ■-^ ' ... 

b, employed part-time (less than 30 hours a week) 



c . 



unemployed 



Wi thou t the explanation of what is considered full- or part-time 
work, the respondents would be likely to have varying interpretations. 
An employee worki/ir; 35 hours a week might consider himself a part- 
time employee if the standards were not given. 
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Example 35 '* 



,Knowlgdge and experience in irious 
agricultural areas, 

(Please place a check in the appro- 
priate column at the right opposite 
each agricultural area listed in 
items a through, f below in order to . 
indicate the scope of the applicant's 
expe:^ience in each, of these areas.) 



0) 

c 
o 



a; 
y 
c 

0) 

u 

0) 



0) 



O 

c 
}^ 

CD 
X 

o 
>\ 



> 



o 
c 



c 

c 



a. Agronomy 

b. Animal husbandry < 
c . Fnro J' r. rv , 

d. Horticulture , . . . 

e. . Range Science,... 

f. Soil Science... .< 



Example 36: Whenever you are given a new assignment, does your 
supervisor tell you in advance enough about it. so 
that you know just how he wants it done? 

a. usual ly 

b. sometimes • , 

c. almost never 



The constructor has included a- qualification in this question. 
He wants to know whether an employee receives adequate instruc tion at 
the time he is given new work, not whether he gets the instruction 
later. 
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So-called "blanket- type" words can cover a .ade range of .r.cerpre 
tations. \vfhat is the meaning of "recently", "i.. the near futu;- ", ■ 
"occasionally", etc. to respondents? For example, a group was :sked 
if they had recently attended a supervisor; training course. l two 
respondents, both of v^hom had attended sue. a course, wi thin th past 
three years,- one would check "yes" while :.ho other would check 
because he did not consider his attendan^-e at the course a "rec 
experience. 



It is sensible to specify the terms in which an answer i. L> 
be given, provided it is possible to do so. Instead of askin: how 
often, an employee performs a task, ask him how many days in t:: 
past iiii^nlb or how many hours a week he spent on that work. 

5. ■ Be cautious about extreme words. 

Words like "always", "never", "anyone", and "nobody" may limit 
the item content so that results are distorted. Their inclusion in 
items should be generally ■ restricted to situations where extreme 
judgements are desired. 

Example 37: Does your : niu diatc - pervis : you clear 

; ■' ins true ; 1 on r, ai) j t your job luci-r.^? 

(~| always 

l~| most of the ri: 

I ] sometimes 

r~j never 

Extremes can- be avoided in some items by using words like 
"generally", "usually", "most", and "practically all." A X 
respondent may. think that a statement is generally accurate^et 
be aware of exceptions to it. Under these ci rcumstances ^iie would 



hesi'tate to express full agreement with the s tatement^bu t would 



agree with a question wording that acknowledged these exceptions 
Avoid negative wording. ^ / 



It is confusing and annoying to a resplendent to have to decide 
what a "yes" or a "no" answer to a nega^i^ely worded item means. 

•Example 38: Is he unable to^Yllow directions? Yes Q No Q 

/ 
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.Checking "Yes" means, "Yes, he is unable to follow directions." 
Checking "No" means, "No, he is not unable (able) to follow directions. 
It would be much easier for the respondent to answer either ot the 
positive reword-ngs of this question, shown in example 39. 



ExuL.ple 39: la he able to follow directions? Yes Q ' No [~] 

He ip (able) to follow directions, 
(unable) 

7. Use proper grammar. 

Item writers should follow the same rules of grammar that, all 
writers should follow. In addition, the item writer, in the interest \_ 
of clarity, should use the simplest possible sentence structure. 
Slang and colloquial terms should be avoided. 

a. 'voi- rdings which may bias response. 

Leading questions have no place in a conscientiously prepared 
questionnaire. . ~ s 

Example UO: You like your job, don't you? Yes O No Q 

In answering the above question, even the most naive respondent^^ 
would realize that, whether or not he could honestly agree to a "yes" 
answer, that was" undoubtedly the one that was wanted. 

In attitude and opinion questionnaires, it is of particular con- 
cern that the question be worded so that both sides of an issue are 
presented fairly in the body of the i tem and in the alternatives. A 
biased response can arise not only from a distorted presentation of 
the issue, but also from the inclusion in the item of words which may 
have an unfortunate emotional tone for respondents. Special care 
should therefore be taken to avoiH using stereotyped words and words 
-that may make subjects feel threatened. An example of the latter is 
a question on which a respondent is asked for an opinion on a proposed 
change - as opposed to the maintenance of t'ne status quo. Most people 
favor continuing a familiar situation rather than adopting a change, 
although they may like the change in the situation once it is accom- 
pli slied. 

Items that request information which may involve a respondent's 
self-esteem or desire for prestige must be carefully worded. Questions 
about matters such as the amount of supervisory responsibility and 
relative dmportance of a position in an organization should be designed 
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so that respondents provide specif^ic informetion (not opinion) about 
the actual number of people they directly and indirectly supervise, 
what supervisory control they have over others, and what grades and 
positions their subordinates hold. Given this inf ormaLi on by . 
supervisors, and specif ic inf ormation by subordinates about how 
they are supervised, a survey conductor can determine the supervisory 
duties of a position objectively. 

Some survey conductors feel that personal questions (e.g., age-, 
income) should not be asked, and i t is essential that, at all times, 
the privacy of the respondent be protected. Frequently, the plans 
for the study permit gross information. For example, it may be 
sufficient to ask an individual whether he is unde?r or over age 30, 
rather than his exact age. At o-ther ;_imes, it is decided, that , if the 
purposes of the survey are adequately stated and the respondents 
realize that the information is needed,, most persons will answer the 
questions willingly. 

9. In dichotomous and multiple-choice items, make aKernatives. 
mutually exclusive and well-defined. 

This point cannot be overemphasized; overlapping and poorly 
def ined,.al ternatives will confuse respondents and lessen the 
reliability and .Validity of items. 

Example 4,i . vsTiich of the following is the most important to you 
in buying a new car? 



a. 


appearance 


b. 


color 


c . 


des ign 


d. 


durabi li ty 


e. 


price 



,To some persons, appearance and color overlap. Design is con-, 
fusing because it may be selected by some as a synonym for appearance 
and by others as a' reference to the . mechanical qualities of -the car.' 

10. In two-choice and '.-^nltiple-choice items, provide all necessary 
alternatives. 

If the topic of the it^jn permits "No opinion", "Bo th" , . "Don » t 
know", or "Other- specif^^-^esponses, these alternatiives should be 
added to the jespon-^ list. If combinatipns of alternatives are 
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essential to cover aP possible responses 5 they .should be included. 
The constructor should remember that the number of alternatives in 
multiple-choice items can be large. It is^his responsibility to 
,s.Gt up a comprehensive list in an organized manner to insure ease 
of response and scoring. • _ 

IK Make^all alternatives grammatically consistent with the item •• 
premise. . 

Each response to dichotomous or multiple-choice items with an 
incomplete premise should form a complete grammatical sentence when 
added to the premise. 

Example 42: I will' accept appointment 

a. anywhere in the Uni ted ' States 

b. in the Washingt on 5 D . C. area onl.y . ' 

c. .outside the Uni ted' States 

. / d. in the following locations only: 



When the premise is a complete que.stion or sentence, the response 
should also be consistent with the premise. 

Example *43: Do you think your supervisor would object if you 

were offered a better job and requested a transfer? 



/ 

/ 

\ 



a. He certainly would. 

^b.'' He probably would. 

. ' , * c. He probably would not. 

d. ■ ^ He certainly^would hot. 

/■ 

Response (.-i. ) in this example means; "I think * he cert.a'inly 
would' object if I wanted to transfer to a better job." Response 
(b. ) is the same except for the substitution of "probably" for 
"certainly." 

. * ■ 

Whenever possible, each response should have parallel form, i,e., 
be in the same mood, contain the same parts of speech, and be equally ^ 
appropriate to the premise. In the following example which illustrates 
this point, tne first part C9ntains some alternatives that do not 
logically follow the premise; the second part shows how such a 
question should be phrased. 

Example 44; . . . because t?hey are; 

• ( ') bulky. . • 

( ) easily broken., 

( ) . explosive \ \ . 

'( ) deteriorate rapidly 



haKardous 

highly inflammable ! 
particularly heavy , : ■ 

must be aged., be f or e ' u s e 

rea'sons othe^r than those shoism -hove (specify) 



■ / 



, because they: . 
are bulky 

are particularly heavy 
are easily broken " 
are hazardous 
are explosive 

are highly inflammable / 
deterioratf-i' rapidly ^ 
must 'be age;d be fore use ^ - i 
..-have characteri.'-tics other^ than the above 
(specify) 

■ ■ . y .■ 

12, List alternatives in certain items in logical^ order . ^.--^"^ , • 

Alternatives are usually listed in^^rder with the.-exti:eme 
choices at the ends of the listyand t^e other choi.c<»'s in between./' 
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Example 45: Always / 

Frequently ^-''^ 
Sometimes' / 
Rarely ,. ' ^^"^ 

Example 46 : ^. ''Strpngly approve 
/ Approve 
/ ^...^ C'UndecidecJ 
X^-'^ y Disapprove' 
^^/^ / Strongly disapprove 

^ivoid over-e laboration , 



. /Elaboration of an item, contrary to its intended purpose of.. 
^prp\/iding greater clarity, may have the reverse effect aVid 'produce 
,Gont radict.loris ,- ' ' • ' 



Example 47: Wliat school subject do you like least-- that is,, 
spend , li tt le . time studying? ' \ 



; Some pe.-sons will spend little time studying a subject they 

dislike but others' will spend more time on such a subject because 
it may be more ^xd'if f icult for them. 
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K. The qu^:^sLionnai VP i tieins should arranged logically witihintihe 
c[Uos I j onn^ii rci , 

Iwery offcn-t should be made to make it as easy as possible for the 
respondent to answer the questions, and one way to do this is to arrange 
the items in a logical and orderly fashion. Insofar as possible, items 
relating to a particular topic should be grouped together. Sometimes, 
several questions can be set up under one Arabic numeral with subletters. 

F. ■ A ^ood item should yield valid and reliable information . 

If all other conditions of a good questionnaire item have been! ful- 
filled, an item will probably provide valid and reliab le, information . 
Regardless of whether an item evaluation is made on the basis of statistical 
metht^dology or on some other basis, the item inus t .f ul f i 1 1 the purpose for 
whicK- it^was designed by providing accurate information relevant to the 
purpose of the study. 

Just as it is not always possible to measure the validity and relia- 
bility of whole questionnaire forms statistically, it is also not always 
possible to evaluate .statistically the validity and reliability of individ- 
ual ques t ionnaire i te.ins . 

. , Whether the items can be objectively evaluated or not is dependent 
upon the purpose of the questionnaire, the types of items used, and the 
content of the items. The constructor has to decide, after considera- 
tion of these factors, whether ..items can be evaluated statistically. 
It i-s impossible to provide' specif ic rules on which he may base this 
decision because each 'ques tionnai re must be considered individually. 
To illustrate, the validity of most items on a questionnaire used to 
establish position standards cannot be determined statistically because 

there- is no appropriate critet^ion with which to correlate the item data.;.... 

Yet if such it.ems serve the purpose of the questionnaire, that is, 
contribute . 3 :>forrnation for standards determination, they can be considered 
vai:d for that purpose. However, the reliability of some of th'^ items on 
such a questionnaire may be. measured statistically by comparing the 
ac^reeinent among respondents with regard to their responses to the items. 
In, addition, if the items are subjectively scored or rated, a measure of 
rater reliability or agreement can be obtained. If there is high rater' 
reliability, different raters will rate the same questions in much the 
same way, or the same rater will rate the same questions consistently 
upon a later rerating. 

There are some items on which the constructor, although he expects 
accurate responses, will not expect agreement ainong respondents.' For 
example, incumbents in different grades of the same position, subject 
to varying degrees and frequency of supervision, will respond differently 
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to tihc same i Loins concerning the amount: and natiure of supervisory cont: rol. 
Assuming that the information they provide is accurate, it will be UvSrful 
in defining differences among grade bevels, and the items will be valid 
and reliable. 
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CHAPTER 7 
iLc^m Review and Tryout 

I i .. 

The ve.vl<L\\^ and tryout of items before compilation of the first draft 
of the questionnaire form are necessary, although often neglected, steps 
in questionnaire design. Regardless of how carefully the item writer 
has applied the basic rules of item writing, he cannot be, sure that the 
items will effectively evoke the desired information until they are 
reviewed and tried out, . ' . 

Review of questionnaire items ■ ' 

Before reviewing an item, a reviewer^ must become familiar with the 
specific objectives of a study, the subject population, and the methods 
of administration and analysis of material. This knowledge is essential 
because an item must be evciluated in terms of how well it meets the basic 
and Specific conditions necessary. 

To insure systematic and complete review, the reviewer should record 
all his comments and suggestions on a card or sheet of paper attached to 
the one containing the proposed item, 

A reviewer evaluates an item from the point of view of subject- 
matter coverage, bcisic measurement principles, and choice of language. 
In his comments, he may question the use-of par t icular words ; he may 
feel that the standards given to respondents are not sufficiently specific 
Further, he may feel that putting the item in a different form would make 
it more Satisfactory, He may also find that a list of alternatives is 
not complete or that an alternative is\ inaccurately stated. The list of 
possible criticisms is endless, 

\ 

A reviewer should, if possible, make "^concre te suggestions for the" 
improvement of an item. When review is completed , a complete writ ten 
record of criticisms and suggestions is avai Table to the writer for use 
in revising the items. The writer may not agre^e. with some of the comments 
or want to acce^. certain suggestions. Where theVe are differences of 
opinion, pretryout of items often provides informativOn which may resolve 
the differences, 

. Below is a checklist which may be used as a guide by Veyiewers, Not 
alTof the point's in the checklist are applicable to all items. It is 
advisable , , however , to check each item against the entire list Sefore 
deciding- which do not apply. 
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Checklist for Review of Items 

1. Item content: 

a. Is the content limited to one topic or idea (not double-barreled)? 

b. Is the topic or idea relevent to the study? 

2. Is the item likely to yield the information the writer intends it to 
yield? (Is it likely to be valid?) 

3. Doeis the form of the item conform with the coding or rating system 
• which will be used? 

a. Is proper provision made for response(s)? 

4. Is the item appropriate for the administrative method which will be. 
used? 

5. Would the item elicit the desired .information more easily or completely 
if written in a different form? Wnich one? 

a. Free-answer? 

b. Short-answer? 

c. Two -choice? 

d. Multiple-choice? 

6. Two-choice items: 

a. Are 'he alternatives clear-cut, separate? 

b. ' Are the alternatives realistic? ' 

c. Are both alternatives stated ■ clear ly? 

d. If one alternative is -implied but not stated, is the fact that 
there are actually two choices clearly evident? 

e. Are the ' al ternat Ives mutually exclusive? 
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f. Is provision made for a possible third response? (As mention(id 
earlier, a third response is not always necessary.) 

1, No opinion 

2, Don' t know 

3, Both 

4. Neither 

5. No oppportunity to observe 

6. Other; Specify - \ 

Multiple-choice icems; 1 

a. Are the alternatives clear-cut, separate? 1 

b. Are the alternatives mutually exclusive? ^ • 1 

c. Are the alternatives accurate? \ 

d. Is the number of alternatives sufficient? 

e. Are ,the numbers and alternatives listed in logical order? 

f. Is provision made for "no response"? (To repeat, provision 
need not always be made, for "no response",). 

1, None of these 

2, No opportunity to observe . 

3. No opinion ^ 

4. Other: Specify ' 

Where necessary, have a frame of reference and a standard of judgment 
been included? \ 

\ 

a. in the directions for the item? \ 

b. in the premise of the item? \ 

c. in the alternatives 
Item wording: 

a. Is the language clear a'ld precise? 
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I s 


it; 


appropriate for the respondent? 


c. 


Is 


it 


grammatical ly correct? 


d. 


Is 


it: 


worded in posi t ive language ? 


e . 


Is 


it 


easy to read? 


f . 


Is 


it 


easy to comple te ? 



First tryout of items ' - . 

A first tryout of items, even before tentative compilation of the 
quisst ionnai re form, can be a very advantageous step in questionnaire ' 
construction. Many gross errors in form and wording show up at this 
stage in spite of a careful review of the proposed items. 

The questionnaire constructor does not normally attempt statist^ical 
evaluation of his questions in a first item tryout. He is interested 
only in an informal trial of the items since he will have a formal, tryout 
of the completed questionnaire form later, . / 

The constructor assembles .the items" that have been written and 
reviewed and prepares tentative directions for administering and/completi 
the questions. He asks a few people, similar to the sample population 
to be used in the study, to answer the items, making it^clear to them 
that the .tryout is for research purposes and that the items may/ give 
them- some difficulty. When respondents similar to the sample subjects 
are unavailable, a constructor may ask his fellow workers to try out 
the items. - ► 

The first tryout may be carried out by the writer personally. He 
either asks the questions in a personal . interview or '^dmini sters the 
questions by the method which will be used in the whole study. Many ' 
survey -conductors prefer tryout in a personal interview because they 
feel that they can learn much from direct> observation. 

In a tryout, a constructor frequently asks the respondent to comment 
either orally or in writing on 'the quesitionnaire items; he is asked, for 
example, to indicate items which ke l?ad difficulty in understanding, or . 
additional alternatives which he feels should be included. He may also 
be asked to suggest change in wording. , ^ 

The first tryout enables the constructor to (1)' find oyt whether 
any of the subject matter he is investigating is inadequately covered by 
the current nunber of items; (2) discover items in which so many responsei 
were recorded in ti'he "Other- - Speci fy'* category that a revision of alterna- 
tive responses must be made; (3) estimate the amount of time- needed to 
complete the finished form; and (4) test: his coding system--some items 
may be as effective and easier to handle in a different form. 



60 



CHAPTER 8 
Designing the Questionnaire Form 

Compil e tent ative directions to respondent s and ad minis trators 

Directions for respondents whieh will appear at ^the beginning of the 
form and which will provide general instructions for the completion of the 
n .les t ionnai re cover such matters as the purpose of the. survey, the sponsor- 
.' ng agency or office, information intended to motivate respondents to 
c complete the form carefully, general directions concerning the method of 
item response, and, when necessary, sample questions. . 

Every survey conductor strives to obtain complete and frank answers 
to every item on his questionnaire. Ic-^is true that the appearance of the 
form^ its length, and its ease of response are h igh ly ■ important . Yet it 
is in the directions to respondents that many questionnai re constructors 
fail. , They may explain sntisf net ori ly the purpose of: the study and its. 
importance to the conductor of it, but if they do not convince the respond- 
ent of the importance ofhis contribution, they will fail. 

First . of all, 'a respondent must be assured of the confidentiality of 
the information he furnishes. Even in questionnaires which need not be 
signed, this assurance mi'st: be given/ 

Second 3 the importance 'of frank, honest answers must be emphc..: i zed . 
For examples suppose. a respondent is asked to evaluate, the job competence 
of employees. The constructor must, first of all, point, out that the value 
of the. findings will be in direct proportion to the accuracy of the informa- 
tion obtained. Equal ly important is the need to make the respondent under- 
stand that the employees themselves will also suffer from faulty ratings. ' 
We mentioned earlier the reluctance of most persons to rate others low on. a 
scale. Yet an employee prompted on the basis of an unreal is tic rating may 
fail miseirably. ,If a constructor can convince respondents that they do- 
employees no favor by, rating them excellent when they are poor or merely 
adequate, or "by rating them n any way other than the way they should be 
rated, be will have done much to assure the accuracy of questionnaire • ■ ' 
resul ts • ' . 

Directions for specific items arid groups of like items are writte'n- 
. now, if th^iswas not done during the construction of . the questions. ,AI1 
directions for item response should be checked for adequacy at this point; 
the constructor may revise the introductory d i rect ions " af ter reviewing 
those for individual .items. 

For group administration" of a questionnaire, special directions for 
the administrators are required to insure controlled administrative, condi- 
tions. These ins true t ions cover the entire process; e.g. the distribution 
of supplies and. forms, introductory .remarks , directions for completing" the 
form, directions for answering any questions asked by the group members, 
and tl^e collection of forms. All remarks which the administrators will, make 
to the group must be written out completely in. the directions. 
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Directions are also required for interviewers. Wlien the survey 
conductor is intarested in standardized or structured interviews, the 
directions shoul^d be very specific and detailed. When the interviewer 
is to be giveri flexibility in conducting the interv^'ew, the directions 
should provide general guidelines for him. Certr.ain language and pro- 
cedures may be suggested, bi.c the interviewer may use his discretion 
in following them. 

The directions for both response and administration are only tenta- 
tive; they may need to be revised after pretesting. 

Design the f i rst form of the questionnaire . . ' , - 

The items which are acceptable after review and tryout and the tenta 
tive directions to respondents make up the questionnaire form. The ques- 
tionnaire assembled for the pretest is designed carefully, although it is 
usually dupl icated . s i mpl y and inexpensively. 

A. The appearance of the form influences- response . 

A respondent will generally.be conscientious about "completing a form 
which has a pleasing appearance. The constructor does not .have to concern 
himself quite so carefully with the appearance of a form to be completed 
by an interviewer, as long as it is clear and easy to follow. 

Appearance is' affected by the method of . form- reproduction . Printed 
and muLtilithed forms are unquestionably better appearing than mimeographed 
..or dittoed ones. Hpwever, because of,, budget limitations or because the 
number of sub jec ts' d oes not justify the employment of more expensive re- 
production methods, questionnaires are .sometimes not reproduced by the most 
desirable methods. 

More important, appearance is affected by • the' layout of the items on 
the form. The general directions should be arranged so that they can be 
easily read and, if necessary, referred to later. Individual items or 
groups of like items should be set off from onie' ano,ther .-by lines or spaces. 
Arrangement, of items in either columns or rows adds" to the neat appearance 
of the form. ^- 

bj Ease o£ response "is an important consideration in questionnaire 
./design. . - 

■ / . ' - ■ ^ 

A neat form is easier for the respondent to complete than one which 
looks' cluttered and disorganized. The grouping of items for which there 
is one set of directions simplifies the respondent's task-. The numbering 
of- items consecutively without repetition of any numbe^r '.permi ts the 
respondent to see at a glance in what order he is to respond to the items. 
It/also- eliminates confusion when items are referred to by number in the 
dl/rections. The respondent's task is simplified, too, 'if there is sufficient 
space on the form to complete write-in questions legibly and completely. / 
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C. The lengCh of thu questionnaire is another ' cons ide ration in 
questionnaire design. 

Some ques t ionnai res such as mail employment ref.erence forms, rely 
for completion on the respondent's voluntary cooperation. These forms 
therefore, cannot he so lengthy that recipients are unwilling Lo take the ' 
time to complete them. When desi'gning such ques t i onnai res , the constr^uctor 
may find that he must limit the number of pages in the form, thus necessi- 
tating the elimination of the least important items. If, as parp^J^ pre-' 
liminary planning, required subject matter was listed in orderxSf priority, 
the tilimination process issimplified.^ 

/' . . 

Questionnaires should be only as long as is necessary to covpr the. 
required subject matter. it is the e'bns tructor ' s re^s^xuis i bi 1 i ty to design 
the form carefullv and concisely even though he is- fairl y certain that the 
respondents will bo conscientious about completitig it. 

■ ■> - ' y' 

It should be remembered that the greatl^r the proportion of objective 
•items on a form, the easier i t "i s to fill out.. Respondents may be more 
willing t.o complete a long ob jec t i ve^^<^'pe questionnaire than they are to 

fill, out one which requires f ai r 1 y^e ta i led answers. 

. . . ' 

D. Provision fdr the scpring or rating of the form should be ' ' 
i nc luded . ' 

In assembling preceded forms, thg constructor makes ^ certai n that:;the 
system of numbering .and lettering tho items and alternatives corre^^^onds 
to the coding system. When codes are to be assigned later, tlie^-cons true tor 
provides enough^s pace next t;o each item for the code to be wr-itten in and 
tabulated witi^iout confusion or error. 

Thfei^most efficient layout from the point of view of coding and extract- 
ing ^{Jerations is to set up the foim so that' the answer spaces are on one 
si^e of a column or page. Some ques t i onnai re' mate ri al does not lend itself 
y€u this kind of layout, but the cons true, t-cnr should try to fit at least part 
of the copy into it. ■■ 



Pretest the first form of .the. ^lJestionnaire 



The pretest provides the first opportunity for the constructor to 
test the adequacy o.f'his survey plan and the effectiveness of the ■ ques ti on- 
naire i ns t rumen t;"' . In the pretest, the first form of the questionnaire is 
given to a sm^ill group of people, as much like, the sample group ^se lected 
for the .full study as possible. The method :of administration is, in 
general, the same as that planned for the complete study, although special 
directions may be given. These directions may, for example, ask respondents 
to record the length of time required for completion of the form, to make 
any comments or suggestions they like relating to it, or to ask the 
administrator questions about individual items. After the questionnaires 
have, been completed by the sample >group, the constructor has the opportunity 
to test rating, coding, and tabulation procedures. 
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The pretest enables the survey conductor to obtain further insight 
into the effectiveness of the questionnaire. He seeks ^ through the nreans 
and sources available to him, a critical reevaluation of the areas covered^ 
as well as of the i terns , nl ternat ives , directions, etc, that will lead to 
necessary and desirable modifications, revi sions , addi tions , or deletions. 
Some of the questions and problems that emerged can be resolved by inter- 
vi'-nv's with members of the pretost group and with tha administrators. Their 
comments relating to arnb igui t ies in the wording of questions or directions, 
as well: as possible procedural or form changes, will help the constructor 
make necessary revisions. Another aspect of the questionnaire that can 
be determined -from the pretest, gri^up is the acceptability of' the form 
^^to?-them; wl'ien completion depends on voluntary cooperation on the part of 
subjects, this information is important,* Time limits are still another 
concern in many questionnaires, and" the pretest will show whether .some 
items should be eliminated in order that the group may finish in the 
allotted time. 

In analyEing the questionnaire data^ the constructor may meavsure the 
reliability and .validi ty of the. form statistically; if so, He should keep 
in mind the possible bia^^.in the pretest ..group because of its small size, » 
and the effect of questionable items, before deciding whether the validity 
and reliability of the form make it acceptable for the study, ■ If no 
statistical evaluation is made, the data are examined for relevance and 
-ccuracy of the information for the purpose of the survey. They are also 
.becked for adequacy of subject-matter coverage; where information is 
lacking or inadequate, additional items must be prepared to cover the 
pertinent topics. 

Item analysis indicates the extent to which the pretest group com- 
pleted the forms. Item omissions may result 'from inadequate directions, 
insufficient standards of judgment, poor item wording, insufficient time, 
or the disinclination of t he ^ responden t s to- provide the desired information. 
Analysis of individual items points up those which need ^ ' : for 
example, items in which an excessive proiw . -n of repU* . ai;t: written 
into, an "other" or "don't know" category will require rewriting. The 
data will also show whether a large proportion of the replies fell into 
one^ or two categories, or whether they were spread over the range of item 
responses , 

Using pretest results as a guide, a- Constructor makes -necessary 
revisions in the • questionnaire items. The revisions may be in item form 
and/or wording. Some alternatives may be rewritten; others, eliminated 
or added. Changes may also be made in the .directions, standards/for 
judgment, and illustrations. Major changes may be necessary i tems 
are rewritten in differe'nf form. If many or extensive revis^ions are 
required, another pretest is necessary. 
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Assemble the final form of the questionnaire 

. Assembly of the final form of the questionnaire and final directions 
for administration and data processing complete the task of questionnaire 
construction. 

The final questionnaire form is usually similar to the first one un- 
less gross inadequacies appeared in the pretest. The final forni includes 
all the acceptable items from the preliminary form, the items revised on 
the basis of pretest results, and all new items. If the principles of 
good layout were followed in the compilation of the preliminary form, the 
changes in layout will probably be minor ones results n?^ frf^ni item revisions; 
addi tions , . and el i nn r - ' nns . Great car must be taken in checking the 
final form for accuracy of directions, and item wording apd numbering. 
The constructor must, in addition, make certain that all changes are 
reflected in the instructions for administration and data processings 

c 

It is noi wi^thin the scope of .this pamphlet to discuss the details 
of the adminis t :a^^ion , dat a processing, summarization of results, and 
evaluation of a questionnaire. These phases of questionnaires are covered 
in a companion publication entitled "Administration, Data Processing, and 
Analysis of Questionnaires/' > 
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